# pranavdhanabalan/kafka architecture

> Architecture diagram of the GitHub repository pranavdhanabalan/kafka, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-02). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/pranavdhanabalan/kafka
- Repository: https://github.com/pranavdhanabalan/kafka

## Overview

Apache Kafka is a distributed event-streaming platform. Its broad product surface includes broker and metadata management, client connectivity, group coordination, storage, stream processing, and data integration. The main user workflow is to configure and start a broker, then use clients to produce or consume events; Kafka also supports administrative and processing workflows. The excerpts show several coordinator and storage responsibilities, but not enough wiring to verify most runtime handoffs, so those links are marked unverified or omitted.

Main entry points: [`core/src/main/scala/kafka/server`](https://github.com/pranavdhanabalan/kafka/tree/trunk/core/src/main/scala/kafka/server) (Kafka Broker), [`clients/src/main/java/org/apache/kafka/common`](https://github.com/pranavdhanabalan/kafka/tree/trunk/clients/src/main/java/org/apache/kafka/common) (Common Primitives), [`group-coordinator/src/main/java/org/apache/kafka/coordinator/group/GroupCoordinatorService.java`](https://github.com/pranavdhanabalan/kafka/blob/trunk/group-coordinator/src/main/java/org/apache/kafka/coordinator/group/GroupCoordinatorService.java) (Group Coordinator), [`share-coordinator/src/main/java/org/apache/kafka/coordinator/share/ShareCoordinatorService.java`](https://github.com/pranavdhanabalan/kafka/blob/trunk/share-coordinator/src/main/java/org/apache/kafka/coordinator/share/ShareCoordinatorService.java) (Share Coordinator)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_clients["Clients and API"]
  node_client_api["Client APIs"]
  node_common["Common Primitives"]
  node_ssl["SSL Factory"]
end

subgraph group_broker["Broker and Metadata"]
  node_broker["Kafka Broker"]
  node_metadata["Metadata Controller"]
  node_storage_tool["Storage Tool<br/>[StorageTool.scala]"]
end

subgraph group_coordination["Group Coordination"]
  node_group_service["Group Coordinator"]
  node_group_state[("Group Metadata")]
  node_share_service["Share Coordinator"]
  node_raft_expiry["Raft Expiration"]
end

subgraph group_storage["Storage"]
  node_remote_manager["Remote Storage API"]
  node_remote_metadata["Remote Segment Metadata"]
  node_remote_loader["Remote Manager Loader"]
end

subgraph group_integrations["Processing and Integration"]
  node_connect["Kafka Connect"]
  node_streams["Kafka Streams"]
end

node_operator(("Kafka Operator"))
node_client_user(("Client Application"))

node_operator -->|"formats storage"| node_storage_tool
node_operator -->|"starts broker"| node_broker
node_client_user -.->|"uses"| node_client_api
node_client_api -.->|"connects to"| node_broker
node_broker -.->|"uses metadata"| node_metadata
node_broker -.->|"coordinates groups"| node_group_service
node_group_service -.->|"manages state"| node_group_state
node_group_service -->|"uses primitives"| node_common
node_share_service -->|"uses primitives"| node_common
node_share_service -.->|"state metadata"| node_remote_metadata
node_remote_manager -->|"uses metadata"| node_remote_metadata
node_remote_loader -->|"loads implementation"| node_remote_manager
node_ssl -->|"uses security types"| node_common
node_client_user -.->|"integrates data"| node_connect
node_client_user -.->|"processes events"| node_streams
node_raft_expiry -->|"uses timeout"| node_common

click node_client_api "https://github.com/pranavdhanabalan/kafka/tree/trunk/clients/src/main/java/org/apache/kafka/clients"
click node_common "https://github.com/pranavdhanabalan/kafka/tree/trunk/clients/src/main/java/org/apache/kafka/common"
click node_ssl "https://github.com/pranavdhanabalan/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/common/security/ssl/DefaultSslEngineFactory.java"
click node_broker "https://github.com/pranavdhanabalan/kafka/tree/trunk/core/src/main/scala/kafka/server"
click node_metadata "https://github.com/pranavdhanabalan/kafka/tree/trunk/metadata/src/main/java/org/apache/kafka/controller"
click node_storage_tool "https://github.com/pranavdhanabalan/kafka/blob/trunk/core/src/main/scala/kafka/tools/StorageTool.scala"
click node_group_service "https://github.com/pranavdhanabalan/kafka/blob/trunk/group-coordinator/src/main/java/org/apache/kafka/coordinator/group/GroupCoordinatorService.java"
click node_group_state "https://github.com/pranavdhanabalan/kafka/blob/trunk/group-coordinator/src/main/java/org/apache/kafka/coordinator/group/GroupMetadataManager.java"
click node_share_service "https://github.com/pranavdhanabalan/kafka/blob/trunk/share-coordinator/src/main/java/org/apache/kafka/coordinator/share/ShareCoordinatorService.java"
click node_raft_expiry "https://github.com/pranavdhanabalan/kafka/blob/trunk/raft/src/main/java/org/apache/kafka/raft/TimingWheelExpirationService.java"
click node_remote_manager "https://github.com/pranavdhanabalan/kafka/blob/trunk/storage/api/src/main/java/org/apache/kafka/server/log/remote/storage/RemoteStorageManager.java"
click node_remote_metadata "https://github.com/pranavdhanabalan/kafka/blob/trunk/storage/api/src/main/java/org/apache/kafka/server/log/remote/storage/RemoteLogSegmentMetadata.java"
click node_remote_loader "https://github.com/pranavdhanabalan/kafka/blob/trunk/storage/src/main/java/org/apache/kafka/server/log/remote/storage/ClassLoaderAwareRemoteStorageManager.java"
click node_connect "https://github.com/pranavdhanabalan/kafka/tree/trunk/connect/runtime/src/main/java/org/apache/kafka/connect/storage"
click node_streams "https://github.com/pranavdhanabalan/kafka/tree/trunk/streams/src/main/java/org/apache/kafka/streams"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_client_api,node_common,node_ssl,node_client_user toneBlue
class node_broker,node_metadata,node_storage_tool toneAmber
class node_group_service,node_group_state,node_share_service,node_raft_expiry toneMint
class node_remote_manager,node_remote_metadata,node_remote_loader toneRose
class node_connect,node_streams,node_operator toneIndigo
```

## Components

### Clients and API

- [Client APIs](https://github.com/pranavdhanabalan/kafka/tree/trunk/clients/src/main/java/org/apache/kafka/clients): `clients/src/main/java/org/apache/kafka/clients`
- [Common Primitives](https://github.com/pranavdhanabalan/kafka/tree/trunk/clients/src/main/java/org/apache/kafka/common): `clients/src/main/java/org/apache/kafka/common`
- [SSL Factory](https://github.com/pranavdhanabalan/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/common/security/ssl/DefaultSslEngineFactory.java): `clients/src/main/java/org/apache/kafka/common/security/ssl/DefaultSslEngineFactory.java`

### Broker and Metadata

- [Kafka Broker](https://github.com/pranavdhanabalan/kafka/tree/trunk/core/src/main/scala/kafka/server): `core/src/main/scala/kafka/server`
- [Metadata Controller](https://github.com/pranavdhanabalan/kafka/tree/trunk/metadata/src/main/java/org/apache/kafka/controller): `metadata/src/main/java/org/apache/kafka/controller`
- [Storage Tool](https://github.com/pranavdhanabalan/kafka/blob/trunk/core/src/main/scala/kafka/tools/StorageTool.scala): `core/src/main/scala/kafka/tools/StorageTool.scala`

### Group Coordination

- [Group Coordinator](https://github.com/pranavdhanabalan/kafka/blob/trunk/group-coordinator/src/main/java/org/apache/kafka/coordinator/group/GroupCoordinatorService.java): `group-coordinator/src/main/java/org/apache/kafka/coordinator/group/GroupCoordinatorService.java`
- [Group Metadata](https://github.com/pranavdhanabalan/kafka/blob/trunk/group-coordinator/src/main/java/org/apache/kafka/coordinator/group/GroupMetadataManager.java): `group-coordinator/src/main/java/org/apache/kafka/coordinator/group/GroupMetadataManager.java`
- [Share Coordinator](https://github.com/pranavdhanabalan/kafka/blob/trunk/share-coordinator/src/main/java/org/apache/kafka/coordinator/share/ShareCoordinatorService.java): `share-coordinator/src/main/java/org/apache/kafka/coordinator/share/ShareCoordinatorService.java`
- [Raft Expiration](https://github.com/pranavdhanabalan/kafka/blob/trunk/raft/src/main/java/org/apache/kafka/raft/TimingWheelExpirationService.java): `raft/src/main/java/org/apache/kafka/raft/TimingWheelExpirationService.java`

### Storage

- [Remote Storage API](https://github.com/pranavdhanabalan/kafka/blob/trunk/storage/api/src/main/java/org/apache/kafka/server/log/remote/storage/RemoteStorageManager.java): `storage/api/src/main/java/org/apache/kafka/server/log/remote/storage/RemoteStorageManager.java`
- [Remote Segment Metadata](https://github.com/pranavdhanabalan/kafka/blob/trunk/storage/api/src/main/java/org/apache/kafka/server/log/remote/storage/RemoteLogSegmentMetadata.java): `storage/api/src/main/java/org/apache/kafka/server/log/remote/storage/RemoteLogSegmentMetadata.java`
- [Remote Manager Loader](https://github.com/pranavdhanabalan/kafka/blob/trunk/storage/src/main/java/org/apache/kafka/server/log/remote/storage/ClassLoaderAwareRemoteStorageManager.java): `storage/src/main/java/org/apache/kafka/server/log/remote/storage/ClassLoaderAwareRemoteStorageManager.java`

### Processing and Integration

- [Kafka Connect](https://github.com/pranavdhanabalan/kafka/tree/trunk/connect/runtime/src/main/java/org/apache/kafka/connect/storage): `connect/runtime/src/main/java/org/apache/kafka/connect/storage`
- [Kafka Streams](https://github.com/pranavdhanabalan/kafka/tree/trunk/streams/src/main/java/org/apache/kafka/streams): `streams/src/main/java/org/apache/kafka/streams`

### Other components

- **Kafka Operator**
- **Client Application**

## Connections

- Kafka Operator → Storage Tool: formats storage (evidence: `README.md`)
- Kafka Operator → Kafka Broker: starts broker (evidence: `README.md`)
- Client Application → Client APIs: uses
- Client APIs → Kafka Broker: connects to
- Kafka Broker → Metadata Controller: uses metadata
- Kafka Broker → Group Coordinator: coordinates groups
- Group Coordinator → Group Metadata: manages state
- Group Coordinator → Common Primitives: uses primitives (evidence: `group-coordinator/src/main/java/org/apache/kafka/coordinator/group/GroupCoordinatorService.java`)
- Share Coordinator → Common Primitives: uses primitives (evidence: `share-coordinator/src/main/java/org/apache/kafka/coordinator/share/ShareCoordinatorService.java`)
- Share Coordinator → Remote Segment Metadata: state metadata
- Remote Storage API → Remote Segment Metadata: uses metadata (evidence: `storage/api/src/main/java/org/apache/kafka/server/log/remote/storage/RemoteStorageManager.java`)
- Remote Manager Loader → Remote Storage API: loads implementation (evidence: `storage/src/main/java/org/apache/kafka/server/log/remote/storage/ClassLoaderAwareRemoteStorageManager.java`)
- SSL Factory → Common Primitives: uses security types (evidence: `clients/src/main/java/org/apache/kafka/common/security/ssl/DefaultSslEngineFactory.java`)
- Client Application → Kafka Connect: integrates data
- Client Application → Kafka Streams: processes events
- Raft Expiration → Common Primitives: uses timeout (evidence: `raft/src/main/java/org/apache/kafka/raft/TimingWheelExpirationService.java`)
