How marsem-sam/build-your-own-kafka works
SimpleKafka is a Kafka-like distributed messaging system. Producers and consumers use a client library to discover cluster metadata and send or fetch messages through brokers. Brokers coordinate broker and topic metadata through ZooKeeper, assign partition leadership, persist partition logs, and replicate messages. The protocol module encodes and decodes client and broker messages. The README also describes controller election, partitioning, replication, and leader failover; unsampled implementation details are represented within the broker subsystem rather than inferred as separate call paths.
Main entry points: SimpleKafkaBroker.java Broker service · SimpleKafkaClient.java Client transport · Protocol.java Wire protocol · BrokerInfo.java Broker metadata
Components 11
Client APIs
- Producer API
com/simplekafka/client/SimpleKafkaProducer.java - Consumer API
com/simplekafka/client/SimpleKafkaConsumer.java - Client transport
com/simplekafka/client/SimpleKafkaClient.java
Broker Runtime
- Broker service
com/simplekafka/broker/SimpleKafkaBroker.java - Broker metadata
com/simplekafka/broker/BrokerInfo.java
Cluster Coordination
- ZooKeeper client
com/simplekafka/broker/ZookeeperClient.java
Protocol And Storage
- Wire protocol
com/simplekafka/broker/Protocol.java - Partition log
com/simplekafka/broker/Partition.java
Other components
- Producer application
- Consumer application
- ZooKeeper
Connections 14
- Producer application → Producer API: sends messages
README.md - Consumer application → Consumer API: reads messages
README.md - Producer API → Client transport: delegates sends
com/simplekafka/client/SimpleKafkaProducer.java - Consumer API → Client transport: polls messages
com/simplekafka/client/SimpleKafkaConsumer.java - Client transport → Wire protocol: encodes requests
com/simplekafka/client/SimpleKafkaClient.java - Client transport → Broker service: connects to
com/simplekafka/client/SimpleKafkaClient.java - Client transport → Broker metadata: loads metadata
com/simplekafka/client/SimpleKafkaClient.java - Broker service → Wire protocol: dispatches requests
com/simplekafka/broker/SimpleKafkaBroker.java - Broker service → Partition log: uses partitions
com/simplekafka/broker/SimpleKafkaBroker.java - Broker service → ZooKeeper client: registers and watches
com/simplekafka/broker/SimpleKafkaBroker.java - ZooKeeper client → ZooKeeper: stores cluster state
com/simplekafka/broker/ZookeeperClient.java - Broker service → Broker metadata: tracks brokers
com/simplekafka/broker/SimpleKafkaBroker.java - Broker service → Broker service: replicates messages
com/simplekafka/broker/SimpleKafkaBroker.java - Wire protocol → Broker metadata: decodes broker data
com/simplekafka/broker/Protocol.java