How zilliztech/vectordbbench works
Project type and purpose
• This is a full-stack Python application that serves two main use cases:
a) A command-line interface (CLI) for running batch or ad-hoc vector database benchmarks.
b) A web-based interactive UI (built with Python front-end components, likely Streamlit) for configuring benchmarks, running tests, and visualizing results.
• Its core function is to orchestrate standardized benchmark cases (insertion, search, filtered search, streaming) against a variety of vectorDB backends, collect metrics (QPS, latency, cost), and present them in reports/leaderboards.
File structure
• Top level
• DevOps/config: .devcontainer/, Dockerfile, Makefile, GitHub workflows
• Project metadata: pyproject.toml, setup/install scripts
• tests/: pytest-based test suite
• vectordb_bench/
• cli/: click-based CLI entrypoints (vectordbbench commands)
• backend/: core orchestration library
– clients/: plugin-style adapters for each vectorDB (Milvus, Redis, Pinecone, etc.)
– cases, runner, assembler, dataset, data_source, filter, task_runner, utils, result_collector
• frontend/: interactive UI components and pages (config forms, charts, filters)
• config-files/: sample YAMLs for single and batch runs
• custom/: user-defined case definitions (custom JSON)
• results/: JSON result archives and leaderboard scripts
Key insights from README
• Benchmark cases are organized into Capacity, Search, Filtered-Search, Streaming, plus custom.
• Clients implement a common VectorDB abstract API via pydantic configs.
• Two frontends: CLI (click) and web UI (custom Python components).
• Config files drive both CLI and UI.
• Results written as JSON and visualized in the UI, plus a public leaderboard page.
How to model this
1. CLI Interface
2. Web UI Interface
3. Core Benchmark Engine
– Configuration Manager (YAML, env vars)
– Task Orchestrator (runner/, assembler.py)
– Dataset Manager (data_source.py, dataset.py)
– Case Library (backend/cases.py)
– Result Collector & Storage (result_collector.py → results/)
4. Plugin Registry
– DB Clients (backend/clients/*) implementing common API
5. External VectorDB Services
– Cloud or self-hosted (Milvus, Pinecone, Redis, etc.)
• CLI/Web UI → pass user parameters to Core Benchmark Engine
• Engine → loads Config → initiates Dataset Manager → feeds data to Task Orchestrator
• Orchestrator → uses Plugin Registry to instantiate specific DB Client → executes benchmark cases against External DB
• DB Clients ↔ External VectorDB (via network calls)
• Orchestrator → sends metrics to Result Collector → serializes to results/ JSON
• UI → reads JSON results → renders charts/leaderboard
• CLI → prints summaries or writes results to disk
c. Architectural patterns / design principles:
• Plugin pattern for database adapters
• Modular separation: CLI vs. UI vs. core engine vs. clients
• Single source of truth for configurations (YAML + pydantic)
• Layered orchestration: assembler → runner → result collector
d. Key technologies / frameworks:
• Python 3.11+, click for CLI, pydantic for config validation
• pytest for testing
• Frontend: Python UI components (likely Streamlit)
• JSON/Parquet for data interchange
• Docker/devcontainer for environment reproducibility