# orbitinghail/graft architecture

> Architecture diagram of the GitHub repository orbitinghail/graft, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-03). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/orbitinghail/graft
- Repository: https://github.com/orbitinghail/graft

## Overview

Based on our analysis of the file tree and README, “Graft” is a Rust-based transactional page storage engine focused on scaling storage performance with lazy partial replication to the edge and leveraging object storage for durability. Its design supports integration with SQLite via a dedicated extension, but it also exposes libraries (like graft-client) for direct Rust integration. Given the split into several “crates” (for client, core logic, protocol definitions, server-side API, SQLite integration, testing, tooling, and tracing), the system embraces a modular architecture that separates concerns among storage engine logic, network communication, and client–server interactions.

##### Main Components
- Client Components:
- Graft Client (crates/graft-client): Provides APIs, network handling, runtime, and storage abstraction. Its internal modules (e.g., metastore, pagestore, runtime storage, network, sync) represent different layers that interact with the server.
- Graft Tools & Test Crates: (graft-tool, graft-test, graft-test-macro) – these support CLI interactions, testing, and workload simulation.
- Core Components:
- Graft Core (crates/graft-core): Contains fundamental building blocks such as page management, hash tables, ID generation (GID) and low-level storage abstractions.
- Graft Proto (crates/graft-proto): Holds protocol buffer definitions that standardize messages between clients and servers.
- Server Components:
- Graft Server (crates/graft-server): This is your backend component that provides API endpoints (for metadata, pagestore, metrics, health, authentication, etc.), exposing binaries for starting metastore, pagestore, or reading storage segments.
- SQLite Integration:
- Graft SQLite Extension (crates/graft-sqlite-extension) and Graft SQLite (crates/graft-sqlite): Allow embedding Graft functionality as an extension to SQLite, enabling easy use via libgraft.
- Tracing and Instrumentation:
- Graft Tracing (crates/graft-tracing): Provides observability within the system.
- External / Deployment Components:
- Deployment configuration files (Dockerfile, deploy/, tasks/) illustrate how the system is containerized and deployed (for instance using Fly.io).
- Datasets and SQL test files (datasets/, tests/sql/) support initialization and verification of the storage engine’s transactional behavior.

##### Relationships and Interactions
- Separation between the client and server. The Graft Client library interacts with the Graft Server using RPCs or API calls defined by the graft-proto definitions.
- The server exposes multiple API endpoints (metastore, pagestore, etc.) that internally rely on core storage logic from Graft Core and utilize modules from Graft Server (for authentication, metrics, task management, and segment management).
- The SQLite integration components interface either directly with the client or with the backend services, allowing SQL queries to be redirected through libgraft.
- The tracing module and tooling components (CLI and tests) are integrated along the data flow to provide visibility and automated verification of system behavior.

##### Architectural Patterns & Design Principles
- Modularization: Each crate represents a modular component making it easier to swap or test parts individually.
- Separation of Concerns: The client, core, server, and integration components are clearly separated. The API layer (graft-server) makes use of orchestration for metastore and pagestore functionality while delegating lower-level storage operations to graft-core.
- Layered Architecture: The protocol (graft-proto) acts as a contract between client and server; core libraries (graft-core) implement business logic; server modules orchestrate requests; and client libraries manage state, caching, and network communication.
- Extensibility: The presence of test-macro, tooling, and SQLite extension crates indicate an architecture that is designed for both local embeddability and integration with broader ecosystems.

##### Relevant Technologies, Frameworks, and Patterns
- Rust and Cargo for building crates.
- Protobuf for protocol messages (as seen from graft-proto and buf YAML files).
- Docker and Fly configurations for containerization and deployment.
- Testing frameworks integrated with the project (e.g., CI workflows in .github/workflows and dedicated test crates).

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Client Components
    subgraph "Client Components"
        GC["Graft Client (metastore, pagestore, runtime, sync)"]:::client
    end

    %% Tools & Testing
    subgraph "Tools & Testing"
        GT["Graft Tool"]:::external
        GTest["Graft Test"]:::external
        GTM["Graft Test Macro"]:::external
    end

    %% Core & Protocol
    subgraph "Core & Protocol"
        GCOR["Graft Core (page management, ID generation)"]:::core
        GProto["Graft Proto (protocol definitions)"]:::core
    end

    %% Server Components
    subgraph "Server Components"
        GS["Graft Server (API endpoints: metastore, pagestore, auth, metrics)"]:::server
    end

    %% SQLite Integration
    subgraph "SQLite Integration"
        GSE["Graft SQLite Extension"]:::integration
        GSQ["Graft SQLite"]:::integration
    end

    %% Observability
    subgraph "Observability"
        GTri["Graft Tracing"]:::observability
    end

    %% Deployment & Testing
    subgraph "Deployment & Testing"
        DConf["Deployment Configurations (Dockerfile, deploy, tasks)"]:::deployment
        DSTest["Datasets & SQL Testing"]:::deployment
    end

    %% Relationships
    GC  -->|"RPC"| GS
    GC  -->|"uses"| GProto
    GS  -->|"uses"| GProto
    GS  -->|"calls"| GCOR

    GSE -->|"integrates"| GC
    GSE -->|"extends"| GS
    GSQ -->|"embeds"| GS

    GT  -->|"CLI/Test"| GC
    GT  -->|"CLI/Test"| GS
    GTest -->|"Testing"| GS
    GTM -->|"macroTesting"| GS

    GTri -->|"monitors"| GS

    DConf -->|"deploys"| GS
    DSTest -->|"validates"| GCOR

    %% Classes
    classDef client fill:#ADD8E6,stroke:#000,stroke-width:2px;
    classDef external fill:#f9c74f,stroke:#000,stroke-width:2px;
    classDef core fill:#90be6d,stroke:#000,stroke-width:2px;
    classDef server fill:#f94144,stroke:#000,stroke-width:2px;
    classDef integration fill:#90e0ef,stroke:#000,stroke-width:2px;
    classDef observability fill:#a9def9,stroke:#000,stroke-width:2px;
    classDef deployment fill:#bdb2ff,stroke:#000,stroke-width:2px;

    %% Click Events
    click GC "https://github.com/orbitinghail/graft/tree/main/crates/graft-client/"
    click GT "https://github.com/orbitinghail/graft/tree/main/crates/graft-tool/"
    click GTest "https://github.com/orbitinghail/graft/tree/main/crates/graft-test/"
    click GTM "https://github.com/orbitinghail/graft/tree/main/crates/graft-test-macro/"
    click GCOR "https://github.com/orbitinghail/graft/tree/main/crates/graft-core/"
    click GProto "https://github.com/orbitinghail/graft/tree/main/crates/graft-proto/"
    click GS "https://github.com/orbitinghail/graft/tree/main/crates/graft-server/"
    click GSE "https://github.com/orbitinghail/graft/tree/main/crates/graft-sqlite-extension/"
    click GSQ "https://github.com/orbitinghail/graft/tree/main/crates/graft-sqlite/"
    click GTri "https://github.com/orbitinghail/graft/tree/main/crates/graft-tracing/"
    click DConf "https://github.com/orbitinghail/graft/tree/main/deploy/"
    click DSTest "https://github.com/orbitinghail/graft/tree/main/datasets/"
```
