How alibaba/zvec works
1. Repository purpose and top-level shape
- Zvec is an in-process vector database: it runs inside the host application rather than as a separate server.
- The codebase is organized into a public API layer, a database/query layer, a core indexing engine, and a low-level performance library (ailego / turbo), with bindings and tools around the edges.
2. Public entry points
- The primary user-facing surfaces are the C API (src/include/zvec/c_api.h, src/binding/c) and the Python package (python/zvec, src/binding/python).
- The Python package wraps collections, schemas, docs, query parameters, and extension functions for embeddings/reranking. It mirrors the C++ object model rather than implementing its own storage engine.
3. Database layer: collections, schema, query, rerank
- src/db is the main database orchestration layer. It defines collections, schema, docs, query objects, status/error handling, and runtime config.
- This layer is what ties together user operations like create/open, insert, update, delete, and query with the underlying index/storage implementations.
- src/db/reranker and Python extensions provide reranking hooks after candidate retrieval.
4. SQL and hybrid query processing
- src/db/sqlengine parses and plans higher-level queries, including full-text search, vector recall, structured filters, ordering, and hybrid combinations.
- The flow is: SQL/query text → parser/analyzer → planner/optimizer → execution plan nodes, which then call into segment and index primitives.
- This is the main boundary where “query intent” is translated into concrete recalls over vectors, inverted indexes, and filters.
5. Indexing architecture: pluggable index families
- src/core/interface and src/core/framework define the common index abstraction: builders, searchers, streamers, reducers, reformers, metrics, storage, and lifecycle hooks.
- Concrete vector index families live under src/core/algorithm, including flat, HNSW, HNSW sparse, HNSW RaBitQ, IVF, DiskANN, and Vamana.
- Each family follows a similar shape: build/train/index, search, and streaming/persistence support, which makes them interchangeable from the higher layers.
6. Full-text and inverted indexing path
- Full-text search is implemented in the DB layer under src/db/index/column/fts_column.
- It includes tokenization pipelines, ANTLR-generated query parsing, posting lists, BM25 scoring, and iterators for conjunction/disjunction/phrase/term matching.
- Separate inverted-column code handles non-vector scalar/text-like filters and acts as part of the hybrid retrieval pipeline.
7. Storage, persistence, and crash recovery
- Persistent data is handled through src/db/index/storage, including memory-backed, mmap-backed, bufferpool-backed, Arrow IPC, Parquet, and chunked file writers/readers.
- WAL support is explicit in src/db/index/storage/wal, with local WAL file implementations for durability and recovery.
- The README’s crash-safety guarantee comes from this layer: writes are logged, and collections can be reopened after failure.
8. Segment model and incremental layout
- src/db/index/segment and related helpers represent the on-disk/in-memory organization of a collection as segments.
- Segments provide the unit for merging, reading, query-time dispatch, and combining results across columns/indexes.
- This is the main data partitioning boundary between collection-level operations and individual index structures.
9. Core math, quantization, and CPU acceleration
- src/ailego and src/turbo provide the performance foundation: vector math, distance matrices, quantizers, SIMD/dispatch implementations, thread pools, file and memory utilities, and CPU feature detection.
- The math subsystem includes multiple distance families: cosine, Euclidean, inner product, MIPS-to-Euclidean transforms, and quantized integer variants.
- A lot of architecture-critical performance comes from runtime dispatch across x86/AVX/AVX512/NEON/SSE and specialized FP16/int8/int4 kernels.
10. Conversion/reformation and mixed precision support
- src/core/quantizer and src/core/metric convert raw user vectors into index-specific internal formats.
- Reformers/converters support binary, half-float, integer quantization, uniform int8, cosine, and MIPS transformations.
- This is the boundary where ingestion data is normalized for the chosen index and metric, and where search-time scoring is aligned with storage format.
11. Plugin and extension points
- src/core/plugin and src/include/zvec/plugin show that some index capabilities are delivered as plugins, not just built-ins.
- The DiskANN plugin is the notable example, matching the repo’s emphasis on pluggable large-scale retrieval backends.
- Extension points are also exposed in Python for external embedding functions and rerankers, which affect query-time behavior but live outside the core engine.
12. Build, packaging, and portability
- The project is CMake-driven, with platform-specific build support for Linux, macOS, Windows, Android, iOS, and RISC-V visible in workflows and scripts.
- Third-party dependencies are vendored under thirdparty and include RocksDB, Arrow, ANTLR, Protobuf, glog/gflags, cppjieba, CRoaring, FastPFOR, RaBitQ, and others.
- This matters architecturally because the repo is designed to produce both native libraries and language bindings from the same core C++ implementation.
13. Validation and operational tooling
- tests mirrors the architecture closely: low-level utility tests, core algorithm tests, DB/segment/storage tests, SQL engine tests, and crash-recovery tests.
- examples demonstrates the public API shape for C and C++, while tools includes benchmarking, local build/recall utilities, and text-to-vector conversion helpers.
- The CI workflows indicate the project is validated across multiple platforms and build targets, reinforcing that portability is part of the architecture.
14. End-to-end data flow to diagram
- Typical flow: application calls Python/C/C++ API → collection/schema/query objects in db → SQL engine or direct index path → segment dispatch → concrete index/searcher/FTS/inverted store → candidate ranking/refinement → results returned to caller.
- Inserts and updates additionally flow through WAL and storage backends before becoming searchable.
- Hybrid queries can fan out across vector indexes, FTS, and filters, then merge/re-rank results in the DB/query layer.
Main entry points: sqlengine SQL engine · collection.cc Collection · segment Segments · algorithm Vector families
Components 23
Public API
C and Python entry points for host applications.
- C API · Primary C surface for embedding zvec in host apps.
src/include/zvec/c_api.h - Python API · Python package wrapping collections, schemas, docs, params, and extensions.
python/zvec
DB Layer
Collection, schema, query, rerank, and SQL orchestration.
- Collection · Collection lifecycle and CRUD entry point for the embedded database.
src/db/collection.cc - Schema & docs · Collection schema, document model, query parameters, and status types.
src/include/zvec/db - DB common · Shared config, status, error, resources, and utility code for the DB layer.
src/db/common - SQL engine · Parses, analyzes, plans, and executes higher-level hybrid queries.
src/db/sqlengine - Reranker · Post-retrieval reranking hook used by DB and Python extensions.
src/db/reranker/reranker.cc
Core Engine
Pluggable indexing, search, storage, and segment management.
- Segments · Collection partitioning unit coordinating reads, merges, and index dispatch.
src/db/index/segment - Vector columns · Vector-column indexing and combined vector retrieval plumbing.
src/db/index/column/vector_column - Full text · Tokenizer, parser, posting lists, BM25, and phrase/term iterators.
src/db/index/column/fts_column - Inverted column · Scalar and text-like inverted indexing path for hybrid filters.
src/db/index/column/inverted_column - Storage & WAL · Forward stores, file formats, and write-ahead logging for durability.
src/db/index/storage - Index framework · Common index interfaces, builders, searchers, streamers, reducers, and lifecycle hooks.
src/core/interface - Framework · Core index orchestration, factory, flow, metadata, mapping, and plugin plumbing.
src/core/framework - Vector families · Built-in vector index families: flat, HNSW, IVF, DiskANN, Vamana, and variants.
src/core/algorithm - Metric & quantize · Vector metric conversion, reforming, and mixed-precision normalization.
src/core/quantizer
Perf Library
Low-level math, quantization, and runtime acceleration.
Integration
Bindings, plugins, examples, and tools around the core.
- Py binding · Python extension module exposing the C++ model layer.
src/binding/python - C binding · C wrapper for the exported API.
src/binding/c - Plugins · Pluggable index backends such as DiskANN exposed through the core plugin layer.
src/core/plugin - Examples · C and C++ examples showing public API and core usage patterns.
examples - Tools · Benchmarking, recall, conversion, and local build utilities.
tools
Connections 33
- Python API → Py binding: wraps · Python package uses the native binding module.
- Py binding → C API: calls · Python binding ultimately exposes the C/C++ entry surface.
- C binding → C API: exports · C binding implements the exported C API.
- C API → Collection: opens · Public operations create or access collections.
- Collection → Schema & docs: uses · Collections rely on schema, docs, params, and status models.
- Collection → DB common: depends · Collection orchestration uses shared DB infrastructure.
- Collection → SQL engine: queries · Collection query APIs delegate to the SQL/query engine.
- Collection → Segments: organizes · Collections are partitioned and served through segments.
- SQL engine → Segments: plans to · Query plans are lowered to segment-level execution.
- SQL engine → Vector columns: recalls · Vector search plans dispatch into vector column indexes.
- SQL engine → Full text: full-text · Text and hybrid queries use the FTS pipeline.
- SQL engine → Inverted column: filters · Structured and scalar filtering is handled by inverted indexes.
- SQL engine → Reranker: re-ranks · Candidate results can be reranked after retrieval.
- Segments → Storage & WAL: persists · Segments are backed by storage and WAL for durability.
- Segments → Vector columns: hosts · Segments contain column-level vector index state.
- Segments → Full text: hosts · Segments contain text index state and postings.
- Segments → Inverted column: hosts · Segments contain inverted/filter state.
- Vector columns → Index framework: implements · Vector column code uses the shared index abstraction.
- Full text → Storage & WAL: stores · FTS indexing uses persistent stores and merge helpers.
- Inverted column → Storage & WAL: stores · Inverted indexes write through persistence and merge paths.
- Index framework → Framework: adapts · Interface types are orchestrated by the framework layer.
- Framework → Vector families: loads · Framework creates and manages concrete vector families.
- Framework → Plugins: extends · Framework can load plugin-provided index capabilities.
- Vector families → Metric & quantize: normalizes · Algorithms rely on metric and quantization conversion.
- Metric & quantize → ailego: uses · Quantization and metrics build on low-level math/runtime support.
- Vector families → ailego: accelerates · Index implementations use SIMD kernels, math, and utilities.
- turbo → ailego: accelerates · Turbo provides specialized kernels alongside the base runtime library.
- Plugins → Vector families: provides · Plugins contribute concrete index families such as DiskANN.
- Python API → Reranker: extends · Python exposes embedding and rerank extension hooks.
- Examples → C API: demonstrates · Examples show the C-facing API shape.
- Examples → Collection: demonstrates · Examples also cover collection and index workflows.
- Tools → SQL engine: benchmarks · Tools exercise and benchmark query execution paths.
- Tools → Vector families: measures · Tools support recall and performance comparison across indexes.