How apache/kafka works
Apache Kafka is a distributed event-streaming platform for producing, consuming, processing, and administering events. The tree and README describe these product capabilities, but the sampled runtime excerpts focus on group coordination, share-group coordination, storage formatting, remote-log storage, and common utilities. The graph preserves Kafka’s major documented product branches while limiting evidenced edges to relationships shown in excerpts or README statements; unsampled subsystem connections are left unverified.
Main entry points: server Kafka broker · GroupCoordinatorService.java Group coordination · ShareCoordinatorService.java Share coordination · NetworkClient.java Client networking
Components 26
Client APIs
- Producer
clients/src/main/java/org/apache/kafka/clients/producer - Consumer
clients/src/main/java/org/apache/kafka/clients/consumer - Admin API
clients/src/main/java/org/apache/kafka/clients/admin - Client networking
clients/src/main/java/org/apache/kafka/clients/NetworkClient.java
Broker Runtime
- Kafka broker
core/src/main/scala/kafka/server - Replication
core/src/main/scala/kafka/server/ReplicaManager.scala - Group coordination
group-coordinator/src/main/java/org/apache/kafka/coordinator/group/GroupCoordinatorService.java - Group metadata
group-coordinator/src/main/java/org/apache/kafka/coordinator/group/GroupMetadataManager.java - Share coordination
share-coordinator/src/main/java/org/apache/kafka/coordinator/share/ShareCoordinatorService.java - Transactions
transaction-coordinator/src/main/java/org/apache/kafka/coordinator/transaction - Remote storage API
storage/api/src/main/java/org/apache/kafka/server/log/remote/storage/RemoteStorageManager.java - Remote storage adapter
storage/src/main/java/org/apache/kafka/server/log/remote/storage/ClassLoaderAwareRemoteStorageManager.java - Segment metadata
storage/api/src/main/java/org/apache/kafka/server/log/remote/storage/RemoteLogSegmentMetadata.java - Remote metadata store
storage/api/src/main/java/org/apache/kafka/server/log/remote/storage
Streaming Extensions
- Kafka Connect
connect/runtime/src/main/java/org/apache/kafka/connect/runtime - Record transforms
connect/transforms/src/main/java/org/apache/kafka/connect/transforms/TimestampRouter.java - Kafka Streams
streams/src/main/java/org/apache/kafka/streams - MirrorMaker client
connect/mirror-client/src/main/java/org/apache/kafka/connect/mirror/MirrorClient.java
Operations
- Storage tool
core/src/main/scala/kafka/tools/StorageTool.scala
Shared Infrastructure
- Metrics
clients/src/main/java/org/apache/kafka/common/metrics/Metrics.java - TLS configuration
clients/src/main/java/org/apache/kafka/common/security/ssl/DefaultSslEngineFactory.java - UUID utilities
clients/src/main/java/org/apache/kafka/common/Uuid.java - Time utilities
clients/src/main/java/org/apache/kafka/common/utils/Time.java - Topic utilities
clients/src/main/java/org/apache/kafka/common/internals/Topic.java - Protocol errors
clients/src/main/java/org/apache/kafka/common/protocol/Errors.java
Other components
- Application user
Connections 29
- Application user → Producer: publishes events
README.md - Application user → Consumer: reads events
README.md - Application user → Admin API: administers cluster
README.md - Producer → Client networking: sends requests
- Consumer → Client networking: sends requests
- Admin API → Client networking: sends requests
- Client networking → Kafka broker: network protocol
- Kafka broker → Replication: hosts
- Kafka broker → Group coordination: coordinates groups
- Group coordination → Group metadata: manages metadata
- Kafka broker → Share coordination: coordinates shares
- Kafka broker → Transactions: coordinates transactions
core/src/main/scala/kafka/server/ReplicaManager.scala - Kafka broker → Remote storage adapter: uses adapter
- Remote storage adapter → Remote storage API: delegates storage
storage/src/main/java/org/apache/kafka/server/log/remote/storage/ClassLoaderAwareRemoteStorageManager.java - Remote storage API → Segment metadata: accepts segment metadata
storage/api/src/main/java/org/apache/kafka/server/log/remote/storage/RemoteStorageManager.java - Remote storage API → Remote metadata store: metadata coordinated separately
storage/api/src/main/java/org/apache/kafka/server/log/remote/storage/RemoteStorageManager.java - Application user → Kafka Connect: integrates data
README.md - Kafka Connect → Record transforms: applies transforms
- Application user → Kafka Streams: builds stream apps
README.md - Application user → MirrorMaker client: mirrors clusters
- Application user → Storage tool: formats storage
core/src/main/scala/kafka/tools/StorageTool.scala - Storage tool → Remote metadata store: formats metadata
- Share coordination → Time utilities: uses time
share-coordinator/src/main/java/org/apache/kafka/coordinator/share/ShareCoordinatorService.java - Group coordination → Topic utilities: uses topic constants
group-coordinator/src/main/java/org/apache/kafka/coordinator/group/GroupCoordinatorService.java - Share coordination → Topic utilities: uses topic constants
share-coordinator/src/main/java/org/apache/kafka/coordinator/share/ShareCoordinatorService.java - Group coordination → Protocol errors: maps protocol errors
group-coordinator/src/main/java/org/apache/kafka/coordinator/group/GroupCoordinatorService.java - Share coordination → Protocol errors: maps protocol errors
share-coordinator/src/main/java/org/apache/kafka/coordinator/share/ShareCoordinatorService.java - Remote storage adapter → Segment metadata: handles segment metadata
storage/src/main/java/org/apache/kafka/server/log/remote/storage/ClassLoaderAwareRemoteStorageManager.java - Application user → TLS configuration: configures secure clients