# v6d-io/v6d architecture

> Architecture diagram of the GitHub repository v6d-io/v6d, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-26). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/v6d-io/v6d
- Repository: https://github.com/v6d-io/v6d

## Overview

##### Project Type and Purpose
- Vineyard is a distributed, in‐memory immutable data manager. It consists of a core C++ server (`vineyardd`), multi‐language client libraries (C++, Python, Go, Java, Rust), pluggable I/O drivers, and a Kubernetes Operator for cloud-native deployment. Its goal is zero-copy data sharing across systems.

##### File Structure Analysis
- Top-level CMake and `src/` folders: build the core engine.
- `src/client` and `src/server`: client–server interaction via IPC/gRPC, memory management, meta services (etcd/Redis).
- Language bindings: `python/`, `java/`, `go/`, `rust/` each implement client SDKs, codegen, tests.
- `modules/`: C++ plugins (basic data structures, graph, fuse, LLM cache, etc.).
- Kubernetes operator in `k8s/`: CRDs, controllers (`controllers/k8s/*`), CLI (`cmd/`), templates (`pkg/templates`), tests.
- Deployment artifacts: `charts/` (Helm), Dockerfiles under `docker/`, Prometheus rules.

##### README Insights
- Vineyard runs as a long-lived daemon that manages shared memory and metadata backed by etcd/Redis.
- Clients across languages connect via IPC sockets or RPC to read/write immutable data structures.
- A Kubernetes Operator watches custom resources (e.g., VineyardCluster, Vineyardd) and uses templated manifests to deploy/scale the core service and sidecars (CSI driver, scheduler plugin, webhooks).

a. Main Components
- Core Service (“vineyardd cluster”):
- IPC Server
- RPC Server
- Memory Manager (shared memory, spill to disk)
- Meta Service (etcd/Redis)
- Client SDKs: C++, Python, Java, Go, Rust
- Modules & Drivers: DS modules (arrays, dataframes, graphs), I/O drivers (Parquet, ORC, S3), stream pipelines
- Kubernetes Operator: CRD Controller, CLI (`vineyardctl`), Templates
- External Services: Kubernetes API, etcd/Redis, Prometheus (metrics), Container Registry (Docker images)

b. Relationships & Interactions
- Clients → IPC/RPC → vineyardd.
- vineyardd ↔ etcd/Redis for metadata.
- vineyardd ↔ Spill storage (disk, object store via IO drivers).
- k8s Operator ↔ Kubernetes API to manage vineyardd Deployments/StatefulSets, CSI driver DaemonSets, Webhook configurations.
- CI/CD pipelines update Docker images → Artifact Hub/Helm → Kubernetes Operator deploys.
- Prometheus scrapes vineyardd metrics; Operator registers CRDs and Helm chart for user.

c. Architectural Patterns & Principles
- Microservice/Operator pattern for Kubernetes integration (CRD-driven).
- Client–Server with language-agnostic RPC and shared memory IPC.
- Plugin architecture: modules loaded at runtime for DS abstractions and drivers.
- Immutable data semantics enabling zero-copy sharing.
- Resource orchestration via declarative manifests (Helm + CRDs).

d. Key Technologies & Frameworks
- Bindings: pybind11 (Python), JNI/FFI (Java), Go modules, Rust crates.
- Kubernetes: controller-runtime (Kubebuilder), client-go, Helm charts, CSI driver.
- CI/CD: GitHub Actions, CircleCI (`.cirrus.yml`), codecov, pre-commit.

##### Essential Details to Capture
- The dual-protocol interface (IPC vs. RPC).
- Memory vs. metadata separation (shared memory pool vs. metadata store).
- Plugin boundaries for modules and drivers.
- Kubernetes CRD-to-Workload reconciliation path.
- Clients’ zero-copy access via shared memory.

## Diagram (Mermaid)

```mermaid
flowchart TB
  %% Clients Layer
  subgraph "Client SDKs"
    direction TB
    CppSDK["C++ SDK"]:::client
    PythonSDK["Python SDK"]:::client
    JavaSDK["Java SDK"]:::client
    GoSDK["Go SDK"]:::client
    RustSDK["Rust SDK"]:::client
  end

  %% Core Engine Layer
  subgraph "Core Engine" 
    direction TB
    Daemon["vineyardd Daemon"]:::core
    IPC["IPC Server"]:::core
    RPC["RPC Server"]:::core
    MM["Memory Manager"]:::core
    Meta["Meta Service (etcd/Redis)"]:::core
  end

  %% Modules & Drivers
  subgraph "Plugins & Drivers"
    direction TB
    DSMods["DS Modules & IO Drivers"]:::module
    PyIODrv["Python IO Drivers"]:::module
  end

  %% Kubernetes Operator Layer
  subgraph "Kubernetes Operator"
    direction TB
    CRD["CRD Definitions"]:::operator
    Controller["CRD Controller"]:::operator
    CLI["vineyardctl CLI"]:::operator
    Templates["Templating Engine"]:::operator
    HelmChart["Helm Chart"]:::operator
  end

  %% External Services
  subgraph "External Services"
    direction TB
    K8sAPI["Kubernetes API"]:::external
    Etcd["etcd"]:::storage
    Redis["Redis"]:::storage
    Disk["Spill Storage"]:::storage
    Prometheus["Prometheus"]:::external
    Registry["Container Registry"]:::external
  end

  %% Connections: Clients to Core
  CppSDK -->|"IPC/RPC"| IPC
  CppSDK -->|"IPC/RPC"| RPC
  PythonSDK -->|"IPC/RPC"| IPC
  PythonSDK -->|"IPC/RPC"| RPC
  JavaSDK -->|"IPC/RPC"| IPC
  JavaSDK -->|"IPC/RPC"| RPC
  GoSDK -->|"IPC/RPC"| IPC
  GoSDK -->|"IPC/RPC"| RPC
  RustSDK -->|"IPC/RPC"| IPC
  RustSDK -->|"IPC/RPC"| RPC

  %% Core internal
  Daemon --> IPC
  Daemon --> RPC
  Daemon --> MM
  Daemon --> Meta

  %% Core to Modules
  Daemon --> DSMods
  Daemon --> PyIODrv

  %% Core to External
  Daemon -->|metadata| Etcd
  Daemon -->|metadata| Redis
  Daemon -->|spill data| Disk
  Daemon -->|metrics| Prometheus

  %% Operator workflow
  CRD --> Controller
  Controller -->|uses| Templates
  Controller -->|deploys| HelmChart
  Controller -->|calls| K8sAPI
  CLI -->|interacts| K8sAPI

  K8sAPI -->|manages| Daemon
  K8sAPI -->|manages| DSMods

  %% External CI/CD (optional)
  Registry -->|hosts images| HelmChart

  %% Click events
  click IPC "https://github.com/v6d-io/v6d/blob/main/src/server/async/ipc_server.cc"
  click RPC "https://github.com/v6d-io/v6d/blob/main/src/server/async/rpc_server.cc"
  click MM "https://github.com/v6d-io/v6d/blob/main/src/server/memory/allocator.cc"
  click Meta "https://github.com/v6d-io/v6d/blob/main/src/server/services/etcd_meta_service.cc"
  click Daemon "https://github.com/v6d-io/v6d/blob/main/src/server/vineyardd.cc"
  click CppSDK "https://github.com/v6d-io/v6d/blob/main/src/client/client.cc"
  click DSMods "https://github.com/v6d-io/v6d/tree/main/modules"
  click PyIODrv "https://github.com/v6d-io/v6d/tree/main/python/vineyard/drivers/io/adaptors"
  click CRD "https://github.com/v6d-io/v6d/tree/main/k8s/apis/k8s/v1alpha1"
  click Controller "https://github.com/v6d-io/v6d/tree/main/k8s/controllers/k8s"
  click CLI "https://github.com/v6d-io/v6d/blob/main/k8s/cmd/main.go"
  click Templates "https://github.com/v6d-io/v6d/tree/main/k8s/pkg/templates"
  click HelmChart "https://github.com/v6d-io/v6d/tree/main/charts/vineyard-operator"

  %% Styles
  classDef core fill:#ADD8E6,stroke:#000,stroke-width:1px
  classDef client fill:#90EE90,stroke:#000,stroke-width:1px
  classDef operator fill:#FFA500,stroke:#000,stroke-width:1px
  classDef external fill:#D3D3D3,stroke:#000,stroke-width:1px
  classDef storage fill:#DEB887,stroke:#000,stroke-width:1px
  classDef module fill:#DDA0DD,stroke:#000,stroke-width:1px
```
