How ceph/ceph-nvmeof works
Project type and purpose
• This is an open-source, containerized NVMe-over-Fabrics (NVMe-oF) Gateway service for Ceph block storage. It exposes RBD images as NVMe-oF namespaces, provides a CLI (via gRPC), a discovery service, and supports monitoring (Prometheus/Grafana).
File structure
• Top-level: Dockerfiles, docker-compose.yaml, Makefile, mk/ (build scripts), .env and config templates.
• control/: Python package implementing CLI (cli.py), server (server.py), gRPC bindings (grpc.py, proto/), cluster and SPDK control (cluster.py, rebalance.py, discovery.py), state management, utilities.
• spdk/: SPDK source (submodule) for NVMe target implementation.
• monitoring/: Grafana dashboard definition.
• tests/: End-to-end HA and functional tests validating gateway, discovery, CLI, state transitions.
• CI configs: .github/workflows, tox.ini, pdm.toml, pyproject.toml.
README for additional insights
• Describes the containerized architecture: four main services in docker-compose (ceph/vstart cluster, nvmeof gateway, nvmeof-cli, discovery).
• Highlights gRPC control API, Prometheus exporter, SPDK BDEV–to–Ceph cluster context mapping strategies.
• Mentions monitoring (port 10008), mTLS, huge-pages setup, mapping strategies.
a. Main components
• Ceph Cluster Container (vstart): RADOS + MON/OSD
• NVMe-oF Gateway Container (SPDK framework + Python service)
• CLI Container (nvmeof-cli) or external CLI via Python wheel
• Discovery Service (optional microservice)
• Prometheus (scraping gateway metrics) + Grafana dashboard
• External NVMe Initiator (iSCSI‐like client)
• Docker network overlay
b. Relationships & interactions
• Gateway ↔ Ceph: SPDK BDEVs mapped to Ceph RBD via librbd
• CLI ↔ Gateway: gRPC calls (control.grpc) to manage subsystems, namespaces, listeners
• Initiator ↔ Gateway: NVMe-oF TCP transport over ports 4420/4421
• Prometheus → Gateway: HTTP scrape on 10008/tcp
• Discovery → Ceph omap via gateway (optional) for NVMe target enumeration
c. Architectural patterns & principles
• Microservices (each Docker container = single responsibility)
• Control plane (gRPC) vs. data plane (SPDK NVMe-oF)
• Plugin/config-driven mapping strategies (INI-file, env vars)
• Separation of concerns: CLI, server, discovery, monitoring
d. Key technologies/frameworks
• Python (pdm/pyproject), gRPC/protobuf
• SPDK (user-space NVMe target), DPDK (huge pages)
• Ceph RBD (librbd), Docker/docker-compose, Makefiles
• Prometheus exporter, Grafana