# zilliztech/vectordbbench architecture

> Architecture diagram of the GitHub repository zilliztech/vectordbbench, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-08-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/zilliztech/vectordbbench
- Repository: https://github.com/zilliztech/vectordbbench

## Overview

##### Project type and purpose
- This is a full-stack Python application that serves two main use cases:
a) A command-line interface (CLI) for running batch or ad-hoc vector database benchmarks.
b) A web-based interactive UI (built with Python front-end components, likely Streamlit) for configuring benchmarks, running tests, and visualizing results.
- Its core function is to orchestrate standardized benchmark cases (insertion, search, filtered search, streaming) against a variety of vectorDB backends, collect metrics (QPS, latency, cost), and present them in reports/leaderboards.

##### File structure
- Top level
- DevOps/config: .devcontainer/, Dockerfile, Makefile, GitHub workflows
- Project metadata: pyproject.toml, setup/install scripts
- tests/: pytest-based test suite
- vectordb_bench/
- cli/: click-based CLI entrypoints (vectordbbench commands)
- backend/: core orchestration library
– clients/: plugin-style adapters for each vectorDB (Milvus, Redis, Pinecone, etc.)
– cases, runner, assembler, dataset, data_source, filter, task_runner, utils, result_collector
- frontend/: interactive UI components and pages (config forms, charts, filters)
- config-files/: sample YAMLs for single and batch runs
- custom/: user-defined case definitions (custom JSON)
- results/: JSON result archives and leaderboard scripts

##### Key insights from README
- Benchmark cases are organized into Capacity, Search, Filtered-Search, Streaming, plus custom.
- Clients implement a common VectorDB abstract API via pydantic configs.
- Two frontends: CLI (click) and web UI (custom Python components).
- Config files drive both CLI and UI.
- Results written as JSON and visualized in the UI, plus a public leaderboard page.

##### How to model this
1. CLI Interface
2. Web UI Interface
3. Core Benchmark Engine
– Configuration Manager (YAML, env vars)
– Task Orchestrator (runner/, assembler.py)
– Dataset Manager (data_source.py, dataset.py)
– Case Library (backend/cases.py)
– Result Collector & Storage (result_collector.py → results/)
4. Plugin Registry
– DB Clients (backend/clients/*) implementing common API
5. External VectorDB Services
– Cloud or self-hosted (Milvus, Pinecone, Redis, etc.)
- CLI/Web UI → pass user parameters to Core Benchmark Engine
- Engine → loads Config → initiates Dataset Manager → feeds data to Task Orchestrator
- Orchestrator → uses Plugin Registry to instantiate specific DB Client → executes benchmark cases against External DB
- DB Clients ↔ External VectorDB (via network calls)
- Orchestrator → sends metrics to Result Collector → serializes to results/ JSON
- UI → reads JSON results → renders charts/leaderboard
- CLI → prints summaries or writes results to disk
c. Architectural patterns / design principles:
- Plugin pattern for database adapters
- Modular separation: CLI vs. UI vs. core engine vs. clients
- Single source of truth for configurations (YAML + pydantic)
- Layered orchestration: assembler → runner → result collector
d. Key technologies / frameworks:
- Python 3.11+, click for CLI, pydantic for config validation
- pytest for testing
- Frontend: Python UI components (likely Streamlit)
- JSON/Parquet for data interchange
- Docker/devcontainer for environment reproducibility

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Presentation Layer
    subgraph "Presentation Layer"
        direction TB
        CLI["CLI Interface"]:::frontend
        WebUI["Web UI Interface"]:::frontend
    end

    %% Core Engine Layer
    subgraph "Core Benchmark Engine"
        direction TB
        ConfigMgr["Configuration Manager"]:::core
        DatasetMgr["Dataset Manager"]:::core
        TaskOrch["Task Orchestrator"]:::core
        CaseLib["Case Library"]:::core
        ResultColl["Result Collector & Storage"]:::core
    end

    %% Plugin Clients Layer
    subgraph "Plugin Registry & Clients"
        direction TB
        PluginReg["Plugin Registry"]:::clients
        Clients["DB Clients"]:::clients
    end

    %% External Systems
    subgraph "External VectorDB Services"
        direction TB
        ExtDB["VectorDB Instances"]:::external
    end

    %% Relationships
    CLI -->|"User params"| ConfigMgr
    WebUI -->|"Form inputs / Config files"| ConfigMgr
    ConfigMgr -->|"YAML config"| DatasetMgr
    DatasetMgr -->|"Parquet dataset"| TaskOrch
    TaskOrch -->|"bench calls"| PluginReg
    PluginReg -->|"instantiate client"| Clients
    Clients -->|"gRPC/REST calls"| ExtDB
    ExtDB -->|"response metrics"| Clients
    Clients -->|"raw metrics"| TaskOrch
    TaskOrch -->|"metrics data"| ResultColl
    ResultColl -->|"JSON results"| WebUI
    ResultColl -->|"summaries"| CLI

    %% Click Events
    click CLI "https://github.com/zilliztech/vectordbbench/tree/main/vectordb_bench/cli/"
    click WebUI "https://github.com/zilliztech/vectordbbench/tree/main/vectordb_bench/frontend/"
    click ConfigMgr "https://github.com/zilliztech/vectordbbench/blob/main/vectordb_bench/models.py"
    click DatasetMgr "https://github.com/zilliztech/vectordbbench/blob/main/vectordb_bench/backend/data_source.py"
    click TaskOrch "https://github.com/zilliztech/vectordbbench/blob/main/vectordb_bench/backend/assembler.py"
    click CaseLib "https://github.com/zilliztech/vectordbbench/blob/main/vectordb_bench/backend/cases.py"
    click ResultColl "https://github.com/zilliztech/vectordbbench/blob/main/vectordb_bench/backend/result_collector.py"
    click PluginReg "https://github.com/zilliztech/vectordbbench/blob/main/vectordb_bench/backend/clients/api.py"
    click Clients "https://github.com/zilliztech/vectordbbench/tree/main/vectordb_bench/backend/clients/milvus/"
    
    %% Styles
    classDef frontend fill:#AEDFF7,stroke:#0066CC,color:#000;
    classDef core fill:#DFF7AE,stroke:#55AA00,color:#000;
    classDef clients fill:#FFE5A8,stroke:#CC8400,color:#000;
    classDef external fill:#E5D6FF,stroke:#884CAB,color:#000;
```
