How buildthingsuseful/build-your-own-kafka works
1. This is a single-module Java/Maven project that implements a Kafka-like system called SimpleKafka. The code is split into two main runtime roles: broker processes and client processes.
2. The broker side lives under com.simplekafka.broker and is the server/control plane of the system. SimpleKafkaBroker is the executable entry point for each broker instance, while BrokerInfo, Partition, and Protocol support broker metadata, partition state, and wire-format handling.
3. ZooKeeper is the coordination boundary. ZookeeperClient is the broker-side integration point used for broker registration, cluster metadata, leader/controller coordination, and handling broker membership changes. The README makes clear that controller election and partition assignment depend on ZooKeeper.
4. Partition is the core storage/state unit on the broker. Conceptually, it represents one topic partition hosted by a broker, where messages are appended and later read by consumers. In the architecture diagram, this is the per-partition data plane endpoint inside each broker.
5. Protocol is the binary request/response boundary between clients and brokers. It is the layer that translates network bytes into broker operations and back again, so it sits between the client library and broker internals.
6. The client side lives under com.simplekafka.client and is split into three layers. SimpleKafkaClient is the low-level connection/request helper, while SimpleKafkaProducer and SimpleKafkaConsumer are higher-level APIs for producing and consuming messages.
7. Message flow is straightforward: producer or consumer CLI/API -> SimpleKafkaClient -> broker Protocol -> broker Partition. The broker persists or serves messages partition-by-partition, and the client talks to a specific broker host/port as shown in the README examples.
8. Cluster topology is manual in the README but logically distributed. Multiple broker processes are started on different ports, and ZooKeeper coordinates their roles. The system is designed around leader/follower behavior for partitions, with one broker acting as leader and others replicating or taking over on failure.
9. The key architectural boundary is control plane vs data plane. ZooKeeper and broker metadata handling form the control plane; request processing, message append/read, and consumer reads form the data plane. This is the main split to show in a diagram.
10. Operationally, the system is run as separate JVM processes via Maven-built classes. mvn clean package produces the jar, then ZooKeeper is started externally, followed by multiple broker processes, then producer/consumer clients connect to a chosen broker endpoint.
11. There are no additional subsystems visible in the tree such as a separate storage package, schema registry, or admin service. The architecture is intentionally compact: coordination, broker runtime, protocol, and client API are the only major components exposed here.
Main entry points: SimpleKafkaBroker.java Broker · Protocol.java Protocol · SimpleKafkaClient.java Client · SimpleKafkaProducer.java Producer
Components 9
Control plane
Cluster coordination and broker membership management
- ZooKeeper · External coordination boundary for broker registration, leader election, and partition assignment
Broker runtime
Broker process, metadata, wire protocol, and partition state
- Broker · Executable broker process started per node
com/simplekafka/broker/SimpleKafkaBroker.java - ZK client · Broker-side integration with ZooKeeper for cluster state and membership changes
com/simplekafka/broker/ZookeeperClient.java - Broker info · Broker identity and cluster metadata used by coordination and routing
com/simplekafka/broker/BrokerInfo.java - Protocol · Binary request/response boundary between clients and broker internals
com/simplekafka/broker/Protocol.java - Partition · Per-topic partition state for appends, reads, and replication handling
com/simplekafka/broker/Partition.java
Client API
Producer, consumer, and low-level client access
- Client · Low-level client helper that opens broker connections and sends requests
com/simplekafka/client/SimpleKafkaClient.java - Producer · High-level API for publishing messages to brokers
com/simplekafka/client/SimpleKafkaProducer.java - Consumer · High-level API for reading messages from brokers
com/simplekafka/client/SimpleKafkaConsumer.java
Connections 13
- Broker → ZK client: registers / coordinates · Broker process uses ZooKeeper integration for cluster membership and control plane duties
- ZK client → ZooKeeper: syncs state · Persists and watches cluster metadata in the external coordination service
- Broker → Broker info: loads metadata · Broker process uses broker identity and cluster metadata
- Broker → Protocol: hosts · Broker exposes the request/response protocol to clients
- Broker → Partition: manages · Broker process owns partition state and message storage
- Protocol → Partition: reads / appends · Protocol translates network requests into partition operations
- Protocol → Broker info: routes via · Protocol and request handling depend on broker metadata for cluster-aware behavior
- Producer → Client: uses · Producer API delegates connection and request handling to the low-level client
- Consumer → Client: uses · Consumer API delegates broker communication to the low-level client
- Client → Protocol: sends requests · Client communicates with broker-side protocol over the network boundary
- Client → Broker: connects to · Clients target a specific broker endpoint
- Producer → Broker: publishes to · Producer traffic ultimately reaches a broker instance through the client layer
- Consumer → Broker: reads from · Consumer traffic ultimately reaches a broker instance through the client layer