# buildthingsuseful/build-your-own-kafka architecture

> Architecture diagram of the GitHub repository buildthingsuseful/build-your-own-kafka, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-07-05). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/buildthingsuseful/build-your-own-kafka
- Repository: https://github.com/buildthingsuseful/build-your-own-kafka

## Overview

1. This is a single-module Java/Maven project that implements a Kafka-like system called SimpleKafka. The code is split into two main runtime roles: broker processes and client processes.

2. The broker side lives under `com.simplekafka.broker` and is the server/control plane of the system. `SimpleKafkaBroker` is the executable entry point for each broker instance, while `BrokerInfo`, `Partition`, and `Protocol` support broker metadata, partition state, and wire-format handling.

3. ZooKeeper is the coordination boundary. `ZookeeperClient` is the broker-side integration point used for broker registration, cluster metadata, leader/controller coordination, and handling broker membership changes. The README makes clear that controller election and partition assignment depend on ZooKeeper.

4. `Partition` is the core storage/state unit on the broker. Conceptually, it represents one topic partition hosted by a broker, where messages are appended and later read by consumers. In the architecture diagram, this is the per-partition data plane endpoint inside each broker.

5. `Protocol` is the binary request/response boundary between clients and brokers. It is the layer that translates network bytes into broker operations and back again, so it sits between the client library and broker internals.

6. The client side lives under `com.simplekafka.client` and is split into three layers. `SimpleKafkaClient` is the low-level connection/request helper, while `SimpleKafkaProducer` and `SimpleKafkaConsumer` are higher-level APIs for producing and consuming messages.

7. Message flow is straightforward: producer or consumer CLI/API -> `SimpleKafkaClient` -> broker `Protocol` -> broker `Partition`. The broker persists or serves messages partition-by-partition, and the client talks to a specific broker host/port as shown in the README examples.

8. Cluster topology is manual in the README but logically distributed. Multiple broker processes are started on different ports, and ZooKeeper coordinates their roles. The system is designed around leader/follower behavior for partitions, with one broker acting as leader and others replicating or taking over on failure.

9. The key architectural boundary is control plane vs data plane. ZooKeeper and broker metadata handling form the control plane; request processing, message append/read, and consumer reads form the data plane. This is the main split to show in a diagram.

10. Operationally, the system is run as separate JVM processes via Maven-built classes. `mvn clean package` produces the jar, then ZooKeeper is started externally, followed by multiple broker processes, then producer/consumer clients connect to a chosen broker endpoint.

11. There are no additional subsystems visible in the tree such as a separate storage package, schema registry, or admin service. The architecture is intentionally compact: coordination, broker runtime, protocol, and client API are the only major components exposed here.

Main entry points: [`com/simplekafka/broker/SimpleKafkaBroker.java`](https://github.com/buildthingsuseful/build-your-own-kafka/blob/main/com/simplekafka/broker/SimpleKafkaBroker.java) (Broker), [`com/simplekafka/broker/Protocol.java`](https://github.com/buildthingsuseful/build-your-own-kafka/blob/main/com/simplekafka/broker/Protocol.java) (Protocol), [`com/simplekafka/client/SimpleKafkaClient.java`](https://github.com/buildthingsuseful/build-your-own-kafka/blob/main/com/simplekafka/client/SimpleKafkaClient.java) (Client), [`com/simplekafka/client/SimpleKafkaProducer.java`](https://github.com/buildthingsuseful/build-your-own-kafka/blob/main/com/simplekafka/client/SimpleKafkaProducer.java) (Producer)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_control["Control plane"]
  node_zookeeper[("ZooKeeper<br/>coordination service")]
end

subgraph group_broker["Broker runtime"]
  node_broker_main["Broker<br/>entrypoint"]
  node_zk_client["ZK client<br/>coordination adapter"]
  node_broker_info["Broker info<br/>metadata model<br/>[BrokerInfo.java]"]
  node_protocol{{"Protocol<br/>[Protocol.java]"}}
  node_partition[("Partition<br/>state unit<br/>[Partition.java]")]
end

subgraph group_client["Client API"]
  node_client_main["Client<br/>connection helper"]
  node_producer["Producer"]
  node_consumer["Consumer"]
end

node_broker_main -->|"registers / coordinates"| node_zk_client
node_zk_client -->|"syncs state"| node_zookeeper
node_broker_main -->|"loads metadata"| node_broker_info
node_broker_main -->|"hosts"| node_protocol
node_broker_main -->|"manages"| node_partition
node_protocol -->|"reads / appends"| node_partition
node_protocol -->|"routes via"| node_broker_info
node_producer -->|"uses"| node_client_main
node_consumer -->|"uses"| node_client_main
node_client_main -->|"sends requests"| node_protocol
node_client_main -->|"connects to"| node_broker_main
node_producer -->|"publishes to"| node_broker_main
node_consumer -->|"reads from"| node_broker_main

click node_broker_main "https://github.com/buildthingsuseful/build-your-own-kafka/blob/main/com/simplekafka/broker/SimpleKafkaBroker.java"
click node_zk_client "https://github.com/buildthingsuseful/build-your-own-kafka/blob/main/com/simplekafka/broker/ZookeeperClient.java"
click node_broker_info "https://github.com/buildthingsuseful/build-your-own-kafka/blob/main/com/simplekafka/broker/BrokerInfo.java"
click node_protocol "https://github.com/buildthingsuseful/build-your-own-kafka/blob/main/com/simplekafka/broker/Protocol.java"
click node_partition "https://github.com/buildthingsuseful/build-your-own-kafka/blob/main/com/simplekafka/broker/Partition.java"
click node_client_main "https://github.com/buildthingsuseful/build-your-own-kafka/blob/main/com/simplekafka/client/SimpleKafkaClient.java"
click node_producer "https://github.com/buildthingsuseful/build-your-own-kafka/blob/main/com/simplekafka/client/SimpleKafkaProducer.java"
click node_consumer "https://github.com/buildthingsuseful/build-your-own-kafka/blob/main/com/simplekafka/client/SimpleKafkaConsumer.java"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_zookeeper toneBlue
class node_broker_main,node_zk_client,node_broker_info,node_protocol,node_partition toneAmber
class node_client_main,node_producer,node_consumer toneMint
```

## Components

### Control plane

Cluster coordination and broker membership management

- **ZooKeeper**: coordination service · External coordination boundary for broker registration, leader election, and partition assignment

### Broker runtime

Broker process, metadata, wire protocol, and partition state

- [Broker](https://github.com/buildthingsuseful/build-your-own-kafka/blob/main/com/simplekafka/broker/SimpleKafkaBroker.java): entrypoint · Executable broker process started per node · `com/simplekafka/broker/SimpleKafkaBroker.java`
- [ZK client](https://github.com/buildthingsuseful/build-your-own-kafka/blob/main/com/simplekafka/broker/ZookeeperClient.java): coordination adapter · Broker-side integration with ZooKeeper for cluster state and membership changes · `com/simplekafka/broker/ZookeeperClient.java`
- [Broker info](https://github.com/buildthingsuseful/build-your-own-kafka/blob/main/com/simplekafka/broker/BrokerInfo.java): metadata model · Broker identity and cluster metadata used by coordination and routing · `com/simplekafka/broker/BrokerInfo.java`
- [Protocol](https://github.com/buildthingsuseful/build-your-own-kafka/blob/main/com/simplekafka/broker/Protocol.java): wire protocol · Binary request/response boundary between clients and broker internals · `com/simplekafka/broker/Protocol.java`
- [Partition](https://github.com/buildthingsuseful/build-your-own-kafka/blob/main/com/simplekafka/broker/Partition.java): state unit · Per-topic partition state for appends, reads, and replication handling · `com/simplekafka/broker/Partition.java`

### Client API

Producer, consumer, and low-level client access

- [Client](https://github.com/buildthingsuseful/build-your-own-kafka/blob/main/com/simplekafka/client/SimpleKafkaClient.java): connection helper · Low-level client helper that opens broker connections and sends requests · `com/simplekafka/client/SimpleKafkaClient.java`
- [Producer](https://github.com/buildthingsuseful/build-your-own-kafka/blob/main/com/simplekafka/client/SimpleKafkaProducer.java): producer API · High-level API for publishing messages to brokers · `com/simplekafka/client/SimpleKafkaProducer.java`
- [Consumer](https://github.com/buildthingsuseful/build-your-own-kafka/blob/main/com/simplekafka/client/SimpleKafkaConsumer.java): consumer API · High-level API for reading messages from brokers · `com/simplekafka/client/SimpleKafkaConsumer.java`

## Connections

- Broker → ZK client: registers / coordinates · Broker process uses ZooKeeper integration for cluster membership and control plane duties
- ZK client → ZooKeeper: syncs state · Persists and watches cluster metadata in the external coordination service
- Broker → Broker info: loads metadata · Broker process uses broker identity and cluster metadata
- Broker → Protocol: hosts · Broker exposes the request/response protocol to clients
- Broker → Partition: manages · Broker process owns partition state and message storage
- Protocol → Partition: reads / appends · Protocol translates network requests into partition operations
- Protocol → Broker info: routes via · Protocol and request handling depend on broker metadata for cluster-aware behavior
- Producer → Client: uses · Producer API delegates connection and request handling to the low-level client
- Consumer → Client: uses · Consumer API delegates broker communication to the low-level client
- Client → Protocol: sends requests · Client communicates with broker-side protocol over the network boundary
- Client → Broker: connects to · Clients target a specific broker endpoint
- Producer → Broker: publishes to · Producer traffic ultimately reaches a broker instance through the client layer
- Consumer → Broker: reads from · Consumer traffic ultimately reaches a broker instance through the client layer
