# apache/kafka architecture

> Architecture diagram of the GitHub repository apache/kafka, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-07). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/apache/kafka
- Repository: https://github.com/apache/kafka
- Video tour (about a minute): https://gitdiagram.com/apache/kafka/video

## Overview

Apache Kafka is a distributed event-streaming platform for producing, consuming, processing, and administering events. The tree and README describe these product capabilities, but the sampled runtime excerpts focus on group coordination, share-group coordination, storage formatting, remote-log storage, and common utilities. The graph preserves Kafka’s major documented product branches while limiting evidenced edges to relationships shown in excerpts or README statements; unsampled subsystem connections are left unverified.

Main entry points: [`core/src/main/scala/kafka/server`](https://github.com/apache/kafka/tree/trunk/core/src/main/scala/kafka/server) (Kafka broker), [`group-coordinator/src/main/java/org/apache/kafka/coordinator/group/GroupCoordinatorService.java`](https://github.com/apache/kafka/blob/trunk/group-coordinator/src/main/java/org/apache/kafka/coordinator/group/GroupCoordinatorService.java) (Group coordination), [`share-coordinator/src/main/java/org/apache/kafka/coordinator/share/ShareCoordinatorService.java`](https://github.com/apache/kafka/blob/trunk/share-coordinator/src/main/java/org/apache/kafka/coordinator/share/ShareCoordinatorService.java) (Share coordination), [`clients/src/main/java/org/apache/kafka/clients/NetworkClient.java`](https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/clients/NetworkClient.java) (Client networking)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_clients["Client APIs"]
  node_producer["Producer"]
  node_consumer["Consumer"]
  node_admin["Admin API"]
  node_clientnet["Client networking<br/>[NetworkClient.java]"]
end

subgraph group_broker["Broker Runtime"]
  node_broker["Kafka broker"]
  node_replication["Replication"]
  node_groupsvc["Group coordination"]
  node_groupstate[("Group metadata")]
  node_sharesvc["Share coordination"]
  node_transaction["Transactions"]
  node_remoteapi["Remote storage API"]
  node_remoteimpl["Remote storage adapter"]
  node_remote_metadata["Segment metadata"]
  node_metadata_store[("Remote metadata store")]
end

subgraph group_streaming["Streaming Extensions"]
  node_connect["Kafka Connect"]
  node_transforms["Record transforms"]
  node_streams["Kafka Streams"]
  node_mirror["MirrorMaker client<br/>[MirrorClient.java]"]
end

subgraph group_operations["Operations"]
  node_storagecli["Storage tool<br/>[StorageTool.scala]"]
end

subgraph group_shared["Shared Infrastructure"]
  node_metrics["Metrics<br/>[Metrics.java]"]
  node_ssl["TLS configuration"]
  node_uuid["UUID utilities<br/>[Uuid.java]"]
  node_time["Time utilities<br/>[Time.java]"]
  node_topic["Topic utilities<br/>[Topic.java]"]
  node_errors["Protocol errors<br/>[Errors.java]"]
end

node_user(("Application user"))

node_user -->|"publishes events"| node_producer
node_user -->|"reads events"| node_consumer
node_user -->|"administers cluster"| node_admin
node_producer -.->|"sends requests"| node_clientnet
node_consumer -.->|"sends requests"| node_clientnet
node_admin -.->|"sends requests"| node_clientnet
node_clientnet -.->|"network protocol"| node_broker
node_broker -.->|"hosts"| node_replication
node_broker -.->|"coordinates groups"| node_groupsvc
node_groupsvc -.->|"manages metadata"| node_groupstate
node_broker -.->|"coordinates shares"| node_sharesvc
node_broker -.->|"coordinates transactions"| node_transaction
node_broker -.->|"uses adapter"| node_remoteimpl
node_remoteimpl -->|"delegates storage"| node_remoteapi
node_remoteapi -->|"accepts segment metadata"| node_remote_metadata
node_remoteapi -->|"metadata coordinated separately"| node_metadata_store
node_user -->|"integrates data"| node_connect
node_connect -.->|"applies transforms"| node_transforms
node_user -->|"builds stream apps"| node_streams
node_user -.->|"mirrors clusters"| node_mirror
node_user -->|"formats storage"| node_storagecli
node_storagecli -.->|"formats metadata"| node_metadata_store
node_sharesvc -->|"uses time"| node_time
node_groupsvc -->|"uses topic constants"| node_topic
node_sharesvc -->|"uses topic constants"| node_topic
node_groupsvc -->|"maps protocol errors"| node_errors
node_sharesvc -->|"maps protocol errors"| node_errors
node_remoteimpl -->|"handles segment metadata"| node_remote_metadata
node_user -.->|"configures secure clients"| node_ssl

click node_producer "https://github.com/apache/kafka/tree/trunk/clients/src/main/java/org/apache/kafka/clients/producer"
click node_consumer "https://github.com/apache/kafka/tree/trunk/clients/src/main/java/org/apache/kafka/clients/consumer"
click node_admin "https://github.com/apache/kafka/tree/trunk/clients/src/main/java/org/apache/kafka/clients/admin"
click node_clientnet "https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/clients/NetworkClient.java"
click node_broker "https://github.com/apache/kafka/tree/trunk/core/src/main/scala/kafka/server"
click node_replication "https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/server/ReplicaManager.scala"
click node_groupsvc "https://github.com/apache/kafka/blob/trunk/group-coordinator/src/main/java/org/apache/kafka/coordinator/group/GroupCoordinatorService.java"
click node_groupstate "https://github.com/apache/kafka/blob/trunk/group-coordinator/src/main/java/org/apache/kafka/coordinator/group/GroupMetadataManager.java"
click node_sharesvc "https://github.com/apache/kafka/blob/trunk/share-coordinator/src/main/java/org/apache/kafka/coordinator/share/ShareCoordinatorService.java"
click node_transaction "https://github.com/apache/kafka/tree/trunk/transaction-coordinator/src/main/java/org/apache/kafka/coordinator/transaction"
click node_remoteapi "https://github.com/apache/kafka/blob/trunk/storage/api/src/main/java/org/apache/kafka/server/log/remote/storage/RemoteStorageManager.java"
click node_remoteimpl "https://github.com/apache/kafka/blob/trunk/storage/src/main/java/org/apache/kafka/server/log/remote/storage/ClassLoaderAwareRemoteStorageManager.java"
click node_connect "https://github.com/apache/kafka/tree/trunk/connect/runtime/src/main/java/org/apache/kafka/connect/runtime"
click node_transforms "https://github.com/apache/kafka/blob/trunk/connect/transforms/src/main/java/org/apache/kafka/connect/transforms/TimestampRouter.java"
click node_streams "https://github.com/apache/kafka/tree/trunk/streams/src/main/java/org/apache/kafka/streams"
click node_mirror "https://github.com/apache/kafka/blob/trunk/connect/mirror-client/src/main/java/org/apache/kafka/connect/mirror/MirrorClient.java"
click node_storagecli "https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/tools/StorageTool.scala"
click node_metrics "https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/common/metrics/Metrics.java"
click node_ssl "https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/common/security/ssl/DefaultSslEngineFactory.java"
click node_uuid "https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/common/Uuid.java"
click node_time "https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/common/utils/Time.java"
click node_topic "https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/common/internals/Topic.java"
click node_errors "https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/common/protocol/Errors.java"
click node_remote_metadata "https://github.com/apache/kafka/blob/trunk/storage/api/src/main/java/org/apache/kafka/server/log/remote/storage/RemoteLogSegmentMetadata.java"
click node_metadata_store "https://github.com/apache/kafka/tree/trunk/storage/api/src/main/java/org/apache/kafka/server/log/remote/storage"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_producer,node_consumer,node_admin,node_clientnet,node_user toneBlue
class node_broker,node_replication,node_groupsvc,node_groupstate,node_sharesvc,node_transaction,node_remoteapi,node_remoteimpl,node_remote_metadata,node_metadata_store toneAmber
class node_connect,node_transforms,node_streams,node_mirror toneMint
class node_storagecli toneRose
class node_metrics,node_ssl,node_uuid,node_time,node_topic,node_errors toneIndigo
```

## Components

### Client APIs

- [Producer](https://github.com/apache/kafka/tree/trunk/clients/src/main/java/org/apache/kafka/clients/producer): `clients/src/main/java/org/apache/kafka/clients/producer`
- [Consumer](https://github.com/apache/kafka/tree/trunk/clients/src/main/java/org/apache/kafka/clients/consumer): `clients/src/main/java/org/apache/kafka/clients/consumer`
- [Admin API](https://github.com/apache/kafka/tree/trunk/clients/src/main/java/org/apache/kafka/clients/admin): `clients/src/main/java/org/apache/kafka/clients/admin`
- [Client networking](https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/clients/NetworkClient.java): `clients/src/main/java/org/apache/kafka/clients/NetworkClient.java`

### Broker Runtime

- [Kafka broker](https://github.com/apache/kafka/tree/trunk/core/src/main/scala/kafka/server): `core/src/main/scala/kafka/server`
- [Replication](https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/server/ReplicaManager.scala): `core/src/main/scala/kafka/server/ReplicaManager.scala`
- [Group coordination](https://github.com/apache/kafka/blob/trunk/group-coordinator/src/main/java/org/apache/kafka/coordinator/group/GroupCoordinatorService.java): `group-coordinator/src/main/java/org/apache/kafka/coordinator/group/GroupCoordinatorService.java`
- [Group metadata](https://github.com/apache/kafka/blob/trunk/group-coordinator/src/main/java/org/apache/kafka/coordinator/group/GroupMetadataManager.java): `group-coordinator/src/main/java/org/apache/kafka/coordinator/group/GroupMetadataManager.java`
- [Share coordination](https://github.com/apache/kafka/blob/trunk/share-coordinator/src/main/java/org/apache/kafka/coordinator/share/ShareCoordinatorService.java): `share-coordinator/src/main/java/org/apache/kafka/coordinator/share/ShareCoordinatorService.java`
- [Transactions](https://github.com/apache/kafka/tree/trunk/transaction-coordinator/src/main/java/org/apache/kafka/coordinator/transaction): `transaction-coordinator/src/main/java/org/apache/kafka/coordinator/transaction`
- [Remote storage API](https://github.com/apache/kafka/blob/trunk/storage/api/src/main/java/org/apache/kafka/server/log/remote/storage/RemoteStorageManager.java): `storage/api/src/main/java/org/apache/kafka/server/log/remote/storage/RemoteStorageManager.java`
- [Remote storage adapter](https://github.com/apache/kafka/blob/trunk/storage/src/main/java/org/apache/kafka/server/log/remote/storage/ClassLoaderAwareRemoteStorageManager.java): `storage/src/main/java/org/apache/kafka/server/log/remote/storage/ClassLoaderAwareRemoteStorageManager.java`
- [Segment metadata](https://github.com/apache/kafka/blob/trunk/storage/api/src/main/java/org/apache/kafka/server/log/remote/storage/RemoteLogSegmentMetadata.java): `storage/api/src/main/java/org/apache/kafka/server/log/remote/storage/RemoteLogSegmentMetadata.java`
- [Remote metadata store](https://github.com/apache/kafka/tree/trunk/storage/api/src/main/java/org/apache/kafka/server/log/remote/storage): `storage/api/src/main/java/org/apache/kafka/server/log/remote/storage`

### Streaming Extensions

- [Kafka Connect](https://github.com/apache/kafka/tree/trunk/connect/runtime/src/main/java/org/apache/kafka/connect/runtime): `connect/runtime/src/main/java/org/apache/kafka/connect/runtime`
- [Record transforms](https://github.com/apache/kafka/blob/trunk/connect/transforms/src/main/java/org/apache/kafka/connect/transforms/TimestampRouter.java): `connect/transforms/src/main/java/org/apache/kafka/connect/transforms/TimestampRouter.java`
- [Kafka Streams](https://github.com/apache/kafka/tree/trunk/streams/src/main/java/org/apache/kafka/streams): `streams/src/main/java/org/apache/kafka/streams`
- [MirrorMaker client](https://github.com/apache/kafka/blob/trunk/connect/mirror-client/src/main/java/org/apache/kafka/connect/mirror/MirrorClient.java): `connect/mirror-client/src/main/java/org/apache/kafka/connect/mirror/MirrorClient.java`

### Operations

- [Storage tool](https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/tools/StorageTool.scala): `core/src/main/scala/kafka/tools/StorageTool.scala`

### Shared Infrastructure

- [Metrics](https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/common/metrics/Metrics.java): `clients/src/main/java/org/apache/kafka/common/metrics/Metrics.java`
- [TLS configuration](https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/common/security/ssl/DefaultSslEngineFactory.java): `clients/src/main/java/org/apache/kafka/common/security/ssl/DefaultSslEngineFactory.java`
- [UUID utilities](https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/common/Uuid.java): `clients/src/main/java/org/apache/kafka/common/Uuid.java`
- [Time utilities](https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/common/utils/Time.java): `clients/src/main/java/org/apache/kafka/common/utils/Time.java`
- [Topic utilities](https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/common/internals/Topic.java): `clients/src/main/java/org/apache/kafka/common/internals/Topic.java`
- [Protocol errors](https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/common/protocol/Errors.java): `clients/src/main/java/org/apache/kafka/common/protocol/Errors.java`

### Other components

- **Application user**

## Connections

- Application user → Producer: publishes events (evidence: `README.md`)
- Application user → Consumer: reads events (evidence: `README.md`)
- Application user → Admin API: administers cluster (evidence: `README.md`)
- Producer → Client networking: sends requests
- Consumer → Client networking: sends requests
- Admin API → Client networking: sends requests
- Client networking → Kafka broker: network protocol
- Kafka broker → Replication: hosts
- Kafka broker → Group coordination: coordinates groups
- Group coordination → Group metadata: manages metadata
- Kafka broker → Share coordination: coordinates shares
- Kafka broker → Transactions: coordinates transactions (evidence: `core/src/main/scala/kafka/server/ReplicaManager.scala`)
- Kafka broker → Remote storage adapter: uses adapter
- Remote storage adapter → Remote storage API: delegates storage (evidence: `storage/src/main/java/org/apache/kafka/server/log/remote/storage/ClassLoaderAwareRemoteStorageManager.java`)
- Remote storage API → Segment metadata: accepts segment metadata (evidence: `storage/api/src/main/java/org/apache/kafka/server/log/remote/storage/RemoteStorageManager.java`)
- Remote storage API → Remote metadata store: metadata coordinated separately (evidence: `storage/api/src/main/java/org/apache/kafka/server/log/remote/storage/RemoteStorageManager.java`)
- Application user → Kafka Connect: integrates data (evidence: `README.md`)
- Kafka Connect → Record transforms: applies transforms
- Application user → Kafka Streams: builds stream apps (evidence: `README.md`)
- Application user → MirrorMaker client: mirrors clusters
- Application user → Storage tool: formats storage (evidence: `core/src/main/scala/kafka/tools/StorageTool.scala`)
- Storage tool → Remote metadata store: formats metadata
- Share coordination → Time utilities: uses time (evidence: `share-coordinator/src/main/java/org/apache/kafka/coordinator/share/ShareCoordinatorService.java`)
- Group coordination → Topic utilities: uses topic constants (evidence: `group-coordinator/src/main/java/org/apache/kafka/coordinator/group/GroupCoordinatorService.java`)
- Share coordination → Topic utilities: uses topic constants (evidence: `share-coordinator/src/main/java/org/apache/kafka/coordinator/share/ShareCoordinatorService.java`)
- Group coordination → Protocol errors: maps protocol errors (evidence: `group-coordinator/src/main/java/org/apache/kafka/coordinator/group/GroupCoordinatorService.java`)
- Share coordination → Protocol errors: maps protocol errors (evidence: `share-coordinator/src/main/java/org/apache/kafka/coordinator/share/ShareCoordinatorService.java`)
- Remote storage adapter → Segment metadata: handles segment metadata (evidence: `storage/src/main/java/org/apache/kafka/server/log/remote/storage/ClassLoaderAwareRemoteStorageManager.java`)
- Application user → TLS configuration: configures secure clients
