How v6d-io/v6d works
Project Type and Purpose
• Vineyard is a distributed, in‐memory immutable data manager. It consists of a core C++ server (vineyardd), multi‐language client libraries (C++, Python, Go, Java, Rust), pluggable I/O drivers, and a Kubernetes Operator for cloud-native deployment. Its goal is zero-copy data sharing across systems.
File Structure Analysis
• Top-level CMake and src/ folders: build the core engine.
• src/client and src/server: client–server interaction via IPC/gRPC, memory management, meta services (etcd/Redis).
• Language bindings: python/, java/, go/, rust/ each implement client SDKs, codegen, tests.
• modules/: C++ plugins (basic data structures, graph, fuse, LLM cache, etc.).
• Kubernetes operator in k8s/: CRDs, controllers (controllers/k8s/*), CLI (cmd/), templates (pkg/templates), tests.
• Deployment artifacts: charts/ (Helm), Dockerfiles under docker/, Prometheus rules.
README Insights
• Vineyard runs as a long-lived daemon that manages shared memory and metadata backed by etcd/Redis.
• Clients across languages connect via IPC sockets or RPC to read/write immutable data structures.
• A Kubernetes Operator watches custom resources (e.g., VineyardCluster, Vineyardd) and uses templated manifests to deploy/scale the core service and sidecars (CSI driver, scheduler plugin, webhooks).
a. Main Components
• Core Service (“vineyardd cluster”):
• IPC Server
• RPC Server
• Memory Manager (shared memory, spill to disk)
• Meta Service (etcd/Redis)
• Client SDKs: C++, Python, Java, Go, Rust
• Modules & Drivers: DS modules (arrays, dataframes, graphs), I/O drivers (Parquet, ORC, S3), stream pipelines
• Kubernetes Operator: CRD Controller, CLI (vineyardctl), Templates
• External Services: Kubernetes API, etcd/Redis, Prometheus (metrics), Container Registry (Docker images)
b. Relationships & Interactions
• Clients → IPC/RPC → vineyardd.
• vineyardd ↔ etcd/Redis for metadata.
• vineyardd ↔ Spill storage (disk, object store via IO drivers).
• k8s Operator ↔ Kubernetes API to manage vineyardd Deployments/StatefulSets, CSI driver DaemonSets, Webhook configurations.
• CI/CD pipelines update Docker images → Artifact Hub/Helm → Kubernetes Operator deploys.
• Prometheus scrapes vineyardd metrics; Operator registers CRDs and Helm chart for user.
c. Architectural Patterns & Principles
• Microservice/Operator pattern for Kubernetes integration (CRD-driven).
• Client–Server with language-agnostic RPC and shared memory IPC.
• Plugin architecture: modules loaded at runtime for DS abstractions and drivers.
• Immutable data semantics enabling zero-copy sharing.
• Resource orchestration via declarative manifests (Helm + CRDs).
d. Key Technologies & Frameworks
• Bindings: pybind11 (Python), JNI/FFI (Java), Go modules, Rust crates.
• Kubernetes: controller-runtime (Kubebuilder), client-go, Helm charts, CSI driver.
• CI/CD: GitHub Actions, CircleCI (.cirrus.yml), codecov, pre-commit.
Essential Details to Capture
• The dual-protocol interface (IPC vs. RPC).
• Memory vs. metadata separation (shared memory pool vs. metadata store).
• Plugin boundaries for modules and drivers.
• Kubernetes CRD-to-Workload reconciliation path.
• Clients’ zero-copy access via shared memory.