# ceph/ceph-nvmeof architecture

> Architecture diagram of the GitHub repository ceph/ceph-nvmeof, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-07-07). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/ceph/ceph-nvmeof
- Repository: https://github.com/ceph/ceph-nvmeof

## Overview

##### Project type and purpose
- This is an open-source, containerized NVMe-over-Fabrics (NVMe-oF) Gateway service for Ceph block storage. It exposes RBD images as NVMe-oF namespaces, provides a CLI (via gRPC), a discovery service, and supports monitoring (Prometheus/Grafana).

##### File structure
- Top-level: Dockerfiles, docker-compose.yaml, Makefile, mk/ (build scripts), `.env` and config templates.
- control/: Python package implementing CLI (`cli.py`), server (`server.py`), gRPC bindings (`grpc.py`, proto/), cluster and SPDK control (`cluster.py`, `rebalance.py`, `discovery.py`), state management, utilities.
- spdk/: SPDK source (submodule) for NVMe target implementation.
- monitoring/: Grafana dashboard definition.
- tests/: End-to-end HA and functional tests validating gateway, discovery, CLI, state transitions.
- CI configs: `.github/workflows`, `tox.ini`, `pdm.toml`, `pyproject.toml`.

##### README for additional insights
- Describes the containerized architecture: four main services in docker-compose (ceph/vstart cluster, nvmeof gateway, nvmeof-cli, discovery).
- Highlights gRPC control API, Prometheus exporter, SPDK BDEV–to–Ceph cluster context mapping strategies.
- Mentions monitoring (port 10008), mTLS, huge-pages setup, mapping strategies.

a. Main components
- Ceph Cluster Container (vstart): RADOS + MON/OSD
- NVMe-oF Gateway Container (SPDK framework + Python service)
- CLI Container (nvmeof-cli) or external CLI via Python wheel
- Discovery Service (optional microservice)
- Prometheus (scraping gateway metrics) + Grafana dashboard
- External NVMe Initiator (iSCSI‐like client)
- Docker network overlay
b. Relationships & interactions
- Gateway ↔ Ceph: SPDK BDEVs mapped to Ceph RBD via librbd
- CLI ↔ Gateway: gRPC calls (`control.grpc`) to manage subsystems, namespaces, listeners
- Initiator ↔ Gateway: NVMe-oF TCP transport over ports 4420/4421
- Prometheus → Gateway: HTTP scrape on 10008/tcp
- Discovery → Ceph omap via gateway (optional) for NVMe target enumeration
c. Architectural patterns & principles
- Microservices (each Docker container = single responsibility)
- Control plane (gRPC) vs. data plane (SPDK NVMe-oF)
- Plugin/config-driven mapping strategies (INI-file, env vars)
- Separation of concerns: CLI, server, discovery, monitoring
d. Key technologies/frameworks
- Python (pdm/pyproject), gRPC/protobuf
- SPDK (user-space NVMe target), DPDK (huge pages)
- Ceph RBD (librbd), Docker/docker-compose, Makefiles
- Prometheus exporter, Grafana

## Diagram (Mermaid)

```mermaid
flowchart TD
    subgraph "Docker Network Overlay"
        subgraph "Ceph Cluster Container"
            Ceph["Ceph Cluster\n(RADOS + MON/OSD)"]:::container_blue
        end
        subgraph "NVMe-oF Gateway Container"
            Gateway_ctrl["Gateway Control Plane\n(Python Service)"]:::control_orange
            SPDK["SPDK NVMe-oF Data Plane"]:::data_green
            Gateway_ctrl -->|"invokes SPDK APIs"| SPDK
        end
        subgraph "CLI Container (nvmeof-cli)"
            CLI["nvmeof-cli (gRPC Client)"]:::control_orange
        end
        subgraph "Discovery Service Container"
            Discovery["Discovery Service"]:::control_orange
        end
    end

    subgraph "External Entities"
        Initiator["External NVMe Initiator"]:::external
        Prometheus["Prometheus"]:::monitoring
        Grafana["Grafana"]:::monitoring
    end

    CLI -->|"gRPC (control.grpc)"| Gateway_ctrl
    Discovery -->|"gRPC / Ceph omap via gateway"| Gateway_ctrl
    Initiator -->|"NVMe-oF TCP (4420/4421)"| SPDK
    Gateway_ctrl -->|"librbd calls"| Ceph
    Prometheus -->|"scrape /metrics (10008)"| Gateway_ctrl
    Grafana -->|"load dashboard"| Prometheus

    subgraph "Ancillary Components"
        Config["Configuration & Env"]:::config_yellow
        Tests["Tests (e2e & functional)"]:::tests
    end

    Config --> Gateway_ctrl
    Config --> Discovery
    Tests --> CLI
    Tests --> Gateway_ctrl

    %% Click Events
    click Ceph "https://github.com/ceph/ceph-nvmeof/blob/devel/docker-compose.yaml"
    click Ceph "https://github.com/ceph/ceph-nvmeof/blob/devel/Dockerfile.ceph"
    click Gateway_ctrl "https://github.com/ceph/ceph-nvmeof/tree/devel/Dockerfile"
    click Gateway_ctrl "https://github.com/ceph/ceph-nvmeof/tree/devel/control/"
    click SPDK "https://github.com/ceph/ceph-nvmeof/tree/devel/spdk/"
    click CLI "https://github.com/ceph/ceph-nvmeof/blob/devel/control/cli.py"
    click CLI "https://github.com/ceph/ceph-nvmeof/blob/devel/control/__main__.py"
    click CLI "https://github.com/ceph/ceph-nvmeof/blob/devel/control/grpc.py"
    click CLI "https://github.com/ceph/ceph-nvmeof/blob/devel/control/proto/gateway.proto"
    click Discovery "https://github.com/ceph/ceph-nvmeof/blob/devel/control/discovery.py"
    click Discovery "https://github.com/ceph/ceph-nvmeof/blob/devel/control/config.py"
    click Gateway_ctrl "https://github.com/ceph/ceph-nvmeof/blob/devel/control/server.py"
    click Gateway_ctrl "https://github.com/ceph/ceph-nvmeof/blob/devel/control/grpc.py"
    click Gateway_ctrl "https://github.com/ceph/ceph-nvmeof/blob/devel/control/proto/gateway.proto"
    click Gateway_ctrl "https://github.com/ceph/ceph-nvmeof/blob/devel/control/proto/monitor.proto"
    click Gateway_ctrl "https://github.com/ceph/ceph-nvmeof/blob/devel/control/prometheus.py"
    click Grafana "https://github.com/ceph/ceph-nvmeof/blob/devel/monitoring/Ceph-NVMe-oF-Gateways-Dashboard.json"
    click Gateway_ctrl "https://github.com/ceph/ceph-nvmeof/blob/devel/control/cluster.py"
    click Gateway_ctrl "https://github.com/ceph/ceph-nvmeof/blob/devel/control/rebalance.py"
    click Gateway_ctrl "https://github.com/ceph/ceph-nvmeof/blob/devel/control/cephutils.py"
    click Gateway_ctrl "https://github.com/ceph/ceph-nvmeof/blob/devel/control/state.py"
    click Gateway_ctrl "https://github.com/ceph/ceph-nvmeof/blob/devel/control/utils.py"
    click Config "https://github.com/ceph/ceph-nvmeof/blob/devel/ceph-nvmeof.conf"
    click Config "https://github.com/ceph/ceph-nvmeof/blob/devel/.env"
    click Config "https://github.com/ceph/ceph-nvmeof/tree/devel/mk/"
    click Tests "https://github.com/ceph/ceph-nvmeof/tree/devel/tests/"

    %% Styles
    classDef container_blue fill:#D0E8FF,stroke:#0366d6,stroke-width:1px;
    classDef data_green fill:#DBFFD6,stroke:#116611,stroke-width:1px;
    classDef control_orange fill:#FFE6CC,stroke:#FF9933,stroke-width:1px;
    classDef monitoring fill:#FFF5BF,stroke:#CC9900,stroke-width:1px;
    classDef config_yellow fill:#FFF5BF,stroke:#CC9900,stroke-width:1px;
    classDef tests fill:#F0F0F0,stroke:#888888,stroke-width:1px;
    classDef external fill:#E8E8E8,stroke:#555555,stroke-width:1px;
```
