# geohot/minikeyvalue architecture

> Architecture diagram of the GitHub repository geohot/minikeyvalue, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-05). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/geohot/minikeyvalue
- Repository: https://github.com/geohot/minikeyvalue

## Overview

##### Project type and purpose
- This is “minikeyvalue,” a lightweight distributed key–value store implemented in Go (~1 000 lines).
- It exposes an HTTP API (including a subset of S3), uses LevelDB for metadata, and leverages stock nginx as the volume (blob) server.
- Primary use cases: scalable storage of 1 MB–1 GB values, replication across machines and disks, rebalance and rebuild operations, production‐ready at comma.ai.

##### File structure
- Top-level:
- src/ — Go source for the mkv binary (main.go, server.go, s3api.go, rebalance.go, rebuild.go, lib.go)
- volume — compiled volume server (nginx wrapper)
- Dockerfile, go.mod/go.sum, requirements.txt — build/deployment configs
- tools/ — bringup/kill scripts, benchmarks (thrasher.go), tests (s3test.py, test.py), helper utilities
- CI config in .github/workflows/test.yml
- mkv CLI supports subcommands: server, rebalance, rebuild

##### README for architectural insights
- Master server (port 3000) handles metadata and coordinates volumes
- Volume servers (ports 3001–3003) are plain nginx processes serving blobs from local disk directories
- Index persisted in LevelDB; rebuild regenerates it from volumes; rebalance redistributes data when volumes change
- Clients interact over HTTP: GET returns 302 redirect to a volume, PUT/DELETE are proxied/blocking operations

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% External Clients & Testing Tools
    subgraph "Clients & Testing Tools"
        direction TB
        S3Test["s3test.py"]:::tools
        TestScript["test.py"]:::tools
        RTest["rtest.sh"]:::tools
        Thrasher["thrasher.go"]:::tools
        Compare["leveldb_compare.go"]:::tools
    end

    %% Master Server
    subgraph "mkv Master Server" 
        direction TB
        Entry["main.go"]:::frontend
        ServerGO["server.go"]:::frontend
        S3API["s3api.go"]:::frontend
        LibGO["lib.go"]:::metadata
        LibTest["lib_test.go"]:::metadata
        Rebalance["rebalance.go"]:::frontend
        Rebuild["rebuild.go"]:::frontend
    end

    %% Storage Layer
    subgraph "Storage Layer"
        direction TB
        LevelDB[(LevelDB)]:::metadata
        subgraph "Volume Servers"
            direction TB
            VolServer["nginx volume/"]:::storage
            FS["Filesystem"]:::storage
        end
    end

    %% Deployment & CI & Utilities
    subgraph "Deployment & CI & Utilities"
        direction TB
        Dockerfile["Dockerfile"]:::deployment
        GoMod["go.mod"]:::deployment
        GoSum["go.sum"]:::deployment
        CI[".github/workflows/test.yml"]:::tools
        Bringup["bringup.sh"]:::tools
        Kill["kill.sh"]:::tools
    end

    %% Data Flows
    S3Test -->|HTTP GET/PUT/DELETE| ServerGO
    TestScript -->|HTTP GET/PUT/DELETE| ServerGO
    RTest -->|HTTP GET/PUT/DELETE| ServerGO
    Thrasher -.->|benchmark| ServerGO

    Entry --> ServerGO
    Entry --> S3API

    ServerGO -->|uses S3 API| S3API
    ServerGO -->|LevelDB read/write| LevelDB
    S3API -->|LevelDB read/write| LevelDB

    ServerGO -->|proxy PUT/DELETE| VolServer
    S3API -->|GET →302 redirect| VolServer
    VolServer -->|store/retrieve blobs| FS

    Rebalance -->|CLI calls| ServerGO
    Rebalance -->|CLI calls| LevelDB
    Rebalance -->|CLI calls| VolServer
    Rebuild -->|CLI calls| ServerGO
    Rebuild -->|CLI calls| LevelDB
    Rebuild -->|CLI calls| VolServer

    %% Deploy & CI flows
    Dockerfile --> Entry
    GoMod --> Entry
    GoSum --> Entry
    CI -.->|runs tests| ServerGO
    Bringup -.->|controls| VolServer
    Kill -.->|controls| VolServer

    %% Click Events
    click S3Test "https://github.com/geohot/minikeyvalue/blob/master/tools/s3test.py"
    click TestScript "https://github.com/geohot/minikeyvalue/blob/master/tools/test.py"
    click RTest "https://github.com/geohot/minikeyvalue/blob/master/tools/rtest.sh"
    click Thrasher "https://github.com/geohot/minikeyvalue/blob/master/tools/thrasher.go"
    click Compare "https://github.com/geohot/minikeyvalue/blob/master/tools/leveldb_compare.go"
    click Entry "https://github.com/geohot/minikeyvalue/blob/master/src/main.go"
    click ServerGO "https://github.com/geohot/minikeyvalue/blob/master/src/server.go"
    click S3API "https://github.com/geohot/minikeyvalue/blob/master/src/s3api.go"
    click LibGO "https://github.com/geohot/minikeyvalue/blob/master/src/lib.go"
    click LibTest "https://github.com/geohot/minikeyvalue/blob/master/src/lib_test.go"
    click Rebalance "https://github.com/geohot/minikeyvalue/blob/master/src/rebalance.go"
    click Rebuild "https://github.com/geohot/minikeyvalue/blob/master/src/rebuild.go"
    click VolServer "https://github.com/geohot/minikeyvalue/tree/master/volume/"
    click Dockerfile "https://github.com/geohot/minikeyvalue/tree/master/Dockerfile"
    click GoMod "https://github.com/geohot/minikeyvalue/blob/master/go.mod"
    click GoSum "https://github.com/geohot/minikeyvalue/blob/master/go.sum"
    click CI "https://github.com/geohot/minikeyvalue/blob/master/.github/workflows/test.yml"
    click Bringup "https://github.com/geohot/minikeyvalue/blob/master/tools/bringup.sh"
    click Kill "https://github.com/geohot/minikeyvalue/blob/master/tools/kill.sh"

    %% Styles
    classDef frontend fill:#e0f7fa,stroke:#006064,stroke-width:1px
    classDef storage fill:#e8f5e9,stroke:#1b5e20,stroke-width:1px
    classDef metadata fill:#fff3e0,stroke:#e65100,stroke-width:1px
    classDef tools fill:#eceff1,stroke:#607d8b,stroke-width:1px
    classDef deployment fill:#f3e5f5,stroke:#4a148c,stroke-width:1px
```
