How pranavdhanabalan/kafka works
Apache Kafka is a distributed event-streaming platform. Its broad product surface includes broker and metadata management, client connectivity, group coordination, storage, stream processing, and data integration. The main user workflow is to configure and start a broker, then use clients to produce or consume events; Kafka also supports administrative and processing workflows. The excerpts show several coordinator and storage responsibilities, but not enough wiring to verify most runtime handoffs, so those links are marked unverified or omitted.
Main entry points: server Kafka Broker · common Common Primitives · GroupCoordinatorService.java Group Coordinator · ShareCoordinatorService.java Share Coordinator
Components 17
Clients and API
- Client APIs
clients/src/main/java/org/apache/kafka/clients - Common Primitives
clients/src/main/java/org/apache/kafka/common - SSL Factory
clients/src/main/java/org/apache/kafka/common/security/ssl/DefaultSslEngineFactory.java
Broker and Metadata
- Kafka Broker
core/src/main/scala/kafka/server - Metadata Controller
metadata/src/main/java/org/apache/kafka/controller - Storage Tool
core/src/main/scala/kafka/tools/StorageTool.scala
Group Coordination
- Group Coordinator
group-coordinator/src/main/java/org/apache/kafka/coordinator/group/GroupCoordinatorService.java - Group Metadata
group-coordinator/src/main/java/org/apache/kafka/coordinator/group/GroupMetadataManager.java - Share Coordinator
share-coordinator/src/main/java/org/apache/kafka/coordinator/share/ShareCoordinatorService.java - Raft Expiration
raft/src/main/java/org/apache/kafka/raft/TimingWheelExpirationService.java
Storage
- Remote Storage API
storage/api/src/main/java/org/apache/kafka/server/log/remote/storage/RemoteStorageManager.java - Remote Segment Metadata
storage/api/src/main/java/org/apache/kafka/server/log/remote/storage/RemoteLogSegmentMetadata.java - Remote Manager Loader
storage/src/main/java/org/apache/kafka/server/log/remote/storage/ClassLoaderAwareRemoteStorageManager.java
Processing and Integration
- Kafka Connect
connect/runtime/src/main/java/org/apache/kafka/connect/storage - Kafka Streams
streams/src/main/java/org/apache/kafka/streams
Other components
- Kafka Operator
- Client Application
Connections 16
- Kafka Operator → Storage Tool: formats storage
README.md - Kafka Operator → Kafka Broker: starts broker
README.md - Client Application → Client APIs: uses
- Client APIs → Kafka Broker: connects to
- Kafka Broker → Metadata Controller: uses metadata
- Kafka Broker → Group Coordinator: coordinates groups
- Group Coordinator → Group Metadata: manages state
- Group Coordinator → Common Primitives: uses primitives
group-coordinator/src/main/java/org/apache/kafka/coordinator/group/GroupCoordinatorService.java - Share Coordinator → Common Primitives: uses primitives
share-coordinator/src/main/java/org/apache/kafka/coordinator/share/ShareCoordinatorService.java - Share Coordinator → Remote Segment Metadata: state metadata
- Remote Storage API → Remote Segment Metadata: uses metadata
storage/api/src/main/java/org/apache/kafka/server/log/remote/storage/RemoteStorageManager.java - Remote Manager Loader → Remote Storage API: loads implementation
storage/src/main/java/org/apache/kafka/server/log/remote/storage/ClassLoaderAwareRemoteStorageManager.java - SSL Factory → Common Primitives: uses security types
clients/src/main/java/org/apache/kafka/common/security/ssl/DefaultSslEngineFactory.java - Client Application → Kafka Connect: integrates data
- Client Application → Kafka Streams: processes events
- Raft Expiration → Common Primitives: uses timeout
raft/src/main/java/org/apache/kafka/raft/TimingWheelExpirationService.java