# marsem-sam/build-your-own-kafka architecture

> Architecture diagram of the GitHub repository marsem-sam/build-your-own-kafka, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-30). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/marsem-sam/build-your-own-kafka
- Repository: https://github.com/marsem-sam/build-your-own-kafka

## Overview

SimpleKafka is a Kafka-like distributed messaging system. Producers and consumers use a client library to discover cluster metadata and send or fetch messages through brokers. Brokers coordinate broker and topic metadata through ZooKeeper, assign partition leadership, persist partition logs, and replicate messages. The protocol module encodes and decodes client and broker messages. The README also describes controller election, partitioning, replication, and leader failover; unsampled implementation details are represented within the broker subsystem rather than inferred as separate call paths.

Main entry points: [`com/simplekafka/broker/SimpleKafkaBroker.java`](https://github.com/marsem-sam/build-your-own-kafka/blob/main/com/simplekafka/broker/SimpleKafkaBroker.java) (Broker service), [`com/simplekafka/client/SimpleKafkaClient.java`](https://github.com/marsem-sam/build-your-own-kafka/blob/main/com/simplekafka/client/SimpleKafkaClient.java) (Client transport), [`com/simplekafka/broker/Protocol.java`](https://github.com/marsem-sam/build-your-own-kafka/blob/main/com/simplekafka/broker/Protocol.java) (Wire protocol), [`com/simplekafka/broker/BrokerInfo.java`](https://github.com/marsem-sam/build-your-own-kafka/blob/main/com/simplekafka/broker/BrokerInfo.java) (Broker metadata)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_client["Client APIs"]
  node_producer_api["Producer API"]
  node_consumer_api["Consumer API"]
  node_client["Client transport"]
end

subgraph group_broker["Broker Runtime"]
  node_broker["Broker service"]
  node_broker_info["Broker metadata<br/>[BrokerInfo.java]"]
end

subgraph group_coordination["Cluster Coordination"]
  node_zk_client["ZooKeeper client"]
end

subgraph group_storage["Protocol And Storage"]
  node_protocol["Wire protocol<br/>[Protocol.java]"]
  node_partition[("Partition log<br/>[Partition.java]")]
end

node_producer_user(("Producer application"))
node_consumer_user(("Consumer application"))
node_zookeeper[("ZooKeeper")]

node_producer_user -->|"sends messages"| node_producer_api
node_consumer_user -->|"reads messages"| node_consumer_api
node_producer_api -->|"delegates sends"| node_client
node_consumer_api -->|"polls messages"| node_client
node_client -->|"encodes requests"| node_protocol
node_client -->|"connects to"| node_broker
node_client -->|"loads metadata"| node_broker_info
node_broker -->|"dispatches requests"| node_protocol
node_broker -->|"uses partitions"| node_partition
node_broker -->|"registers and watches"| node_zk_client
node_zk_client -->|"stores cluster state"| node_zookeeper
node_broker -->|"tracks brokers"| node_broker_info
node_broker -->|"replicates messages"| node_broker
node_protocol -->|"decodes broker data"| node_broker_info

click node_producer_api "https://github.com/marsem-sam/build-your-own-kafka/blob/main/com/simplekafka/client/SimpleKafkaProducer.java"
click node_consumer_api "https://github.com/marsem-sam/build-your-own-kafka/blob/main/com/simplekafka/client/SimpleKafkaConsumer.java"
click node_client "https://github.com/marsem-sam/build-your-own-kafka/blob/main/com/simplekafka/client/SimpleKafkaClient.java"
click node_broker "https://github.com/marsem-sam/build-your-own-kafka/blob/main/com/simplekafka/broker/SimpleKafkaBroker.java"
click node_broker_info "https://github.com/marsem-sam/build-your-own-kafka/blob/main/com/simplekafka/broker/BrokerInfo.java"
click node_zk_client "https://github.com/marsem-sam/build-your-own-kafka/blob/main/com/simplekafka/broker/ZookeeperClient.java"
click node_protocol "https://github.com/marsem-sam/build-your-own-kafka/blob/main/com/simplekafka/broker/Protocol.java"
click node_partition "https://github.com/marsem-sam/build-your-own-kafka/blob/main/com/simplekafka/broker/Partition.java"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_producer_api,node_consumer_api,node_client toneBlue
class node_broker,node_broker_info,node_zookeeper toneAmber
class node_zk_client toneMint
class node_protocol,node_partition toneRose
class node_producer_user,node_consumer_user toneIndigo
```

## Components

### Client APIs

- [Producer API](https://github.com/marsem-sam/build-your-own-kafka/blob/main/com/simplekafka/client/SimpleKafkaProducer.java): `com/simplekafka/client/SimpleKafkaProducer.java`
- [Consumer API](https://github.com/marsem-sam/build-your-own-kafka/blob/main/com/simplekafka/client/SimpleKafkaConsumer.java): `com/simplekafka/client/SimpleKafkaConsumer.java`
- [Client transport](https://github.com/marsem-sam/build-your-own-kafka/blob/main/com/simplekafka/client/SimpleKafkaClient.java): `com/simplekafka/client/SimpleKafkaClient.java`

### Broker Runtime

- [Broker service](https://github.com/marsem-sam/build-your-own-kafka/blob/main/com/simplekafka/broker/SimpleKafkaBroker.java): `com/simplekafka/broker/SimpleKafkaBroker.java`
- [Broker metadata](https://github.com/marsem-sam/build-your-own-kafka/blob/main/com/simplekafka/broker/BrokerInfo.java): `com/simplekafka/broker/BrokerInfo.java`

### Cluster Coordination

- [ZooKeeper client](https://github.com/marsem-sam/build-your-own-kafka/blob/main/com/simplekafka/broker/ZookeeperClient.java): `com/simplekafka/broker/ZookeeperClient.java`

### Protocol And Storage

- [Wire protocol](https://github.com/marsem-sam/build-your-own-kafka/blob/main/com/simplekafka/broker/Protocol.java): `com/simplekafka/broker/Protocol.java`
- [Partition log](https://github.com/marsem-sam/build-your-own-kafka/blob/main/com/simplekafka/broker/Partition.java): `com/simplekafka/broker/Partition.java`

### Other components

- **Producer application**
- **Consumer application**
- **ZooKeeper**

## Connections

- Producer application → Producer API: sends messages (evidence: `README.md`)
- Consumer application → Consumer API: reads messages (evidence: `README.md`)
- Producer API → Client transport: delegates sends (evidence: `com/simplekafka/client/SimpleKafkaProducer.java`)
- Consumer API → Client transport: polls messages (evidence: `com/simplekafka/client/SimpleKafkaConsumer.java`)
- Client transport → Wire protocol: encodes requests (evidence: `com/simplekafka/client/SimpleKafkaClient.java`)
- Client transport → Broker service: connects to (evidence: `com/simplekafka/client/SimpleKafkaClient.java`)
- Client transport → Broker metadata: loads metadata (evidence: `com/simplekafka/client/SimpleKafkaClient.java`)
- Broker service → Wire protocol: dispatches requests (evidence: `com/simplekafka/broker/SimpleKafkaBroker.java`)
- Broker service → Partition log: uses partitions (evidence: `com/simplekafka/broker/SimpleKafkaBroker.java`)
- Broker service → ZooKeeper client: registers and watches (evidence: `com/simplekafka/broker/SimpleKafkaBroker.java`)
- ZooKeeper client → ZooKeeper: stores cluster state (evidence: `com/simplekafka/broker/ZookeeperClient.java`)
- Broker service → Broker metadata: tracks brokers (evidence: `com/simplekafka/broker/SimpleKafkaBroker.java`)
- Broker service → Broker service: replicates messages (evidence: `com/simplekafka/broker/SimpleKafkaBroker.java`)
- Wire protocol → Broker metadata: decodes broker data (evidence: `com/simplekafka/broker/Protocol.java`)