# dpkp/kafka-python architecture

> Architecture diagram of the GitHub repository dpkp/kafka-python, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-09). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/dpkp/kafka-python
- Repository: https://github.com/dpkp/kafka-python

## Overview

kafka-python is a Python Apache Kafka client. Its principal workflow is applications producing serialized records to Kafka and consumers fetching, decoding, and returning records; grouped consumers additionally coordinate partition assignment and offset commits. The library also provides cluster administration through a client and CLI. Shared networking, metadata, protocol, record, metrics, and error components support these workflows. README statements establish the product capabilities; source-index references support several internal links, while unsampled modules are represented at their existing paths without claiming their internals were inspected.

Main entry points: [`kafka/producer/kafka.py`](https://github.com/dpkp/kafka-python/blob/master/kafka/producer/kafka.py) (Producer API), [`kafka/consumer/group.py`](https://github.com/dpkp/kafka-python/blob/master/kafka/consumer/group.py) (Consumer API), [`kafka/net/manager.py`](https://github.com/dpkp/kafka-python/blob/master/kafka/net/manager.py) (Network manager), [`kafka/admin/client.py`](https://github.com/dpkp/kafka-python/blob/master/kafka/admin/client.py) (Admin client)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_prod["Produce"]
  node_producer["Producer API<br/>[kafka.py]"]
  node_accum["Record batching"]
  node_sender["Network sender<br/>[sender.py]"]
  node_txn["Transactions"]
  node_partitioner["Partitioning"]
end

subgraph group_cons["Consume"]
  node_consumer["Consumer API<br/>[group.py]"]
  node_fetcher["Fetching records<br/>[fetcher.py]"]
  node_subscriptions["Subscriptions"]
  node_coord["Group coordination<br/>[consumer.py]"]
  node_assignors["Partition assignors"]
end

subgraph group_admin["Administration"]
  node_admincli["Admin CLI<br/>[__init__.py]"]
  node_adminclient["Admin client<br/>[client.py]"]
  node_adminops["Admin operations"]
end

subgraph group_shared["Shared client runtime"]
  node_cluster["Cluster metadata<br/>[cluster.py]"]
  node_network["Network manager<br/>[manager.py]"]
  node_backend["Network backends"]
  node_metrics["Metrics"]
end

subgraph group_proto["Wire and records"]
  node_protocol["Protocol encoding"]
  node_records["Record formats<br/>[abc.py]"]
  node_codec["Compression codecs<br/>[codec.py]"]
end

node_app(("Application"))
node_broker[("Kafka brokers")]
node_result["Consumer records"]

node_app -->|"sends records"| node_producer
node_producer -->|"batches records"| node_accum
node_producer -->|"chooses partitions"| node_partitioner
node_producer -->|"dispatches batches"| node_sender
node_producer -->|"manages transactions"| node_txn
node_sender -.->|"sends requests"| node_network
node_app -->|"requests records"| node_consumer
node_consumer -->|"tracks assignments"| node_subscriptions
node_consumer -->|"coordinates groups"| node_coord
node_consumer -->|"fetches records"| node_fetcher
node_coord -->|"selects assignor"| node_assignors
node_fetcher -->|"decodes batches"| node_records
node_fetcher -->|"returns records"| node_result
node_app -->|"runs commands"| node_admincli
node_admincli -->|"invokes client"| node_adminclient
node_adminclient -->|"delegates operations"| node_adminops
node_adminclient -->|"sends requests"| node_network
node_producer -->|"uses networking"| node_network
node_consumer -->|"uses networking"| node_network
node_network -->|"updates metadata"| node_cluster
node_network -->|"runs I/O"| node_backend
node_network -.->|"exchanges messages"| node_protocol
node_protocol -.->|"encodes record data"| node_records
node_records -->|"uses compression"| node_codec
node_producer -->|"produces messages"| node_broker
node_broker -->|"provides messages"| node_consumer
node_producer -->|"exposes metrics"| node_metrics
node_consumer -->|"exposes metrics"| node_metrics
node_adminclient -->|"administers cluster"| node_broker

click node_producer "https://github.com/dpkp/kafka-python/blob/master/kafka/producer/kafka.py"
click node_accum "https://github.com/dpkp/kafka-python/blob/master/kafka/producer/record_accumulator.py"
click node_sender "https://github.com/dpkp/kafka-python/blob/master/kafka/producer/sender.py"
click node_txn "https://github.com/dpkp/kafka-python/blob/master/kafka/producer/transaction_manager.py"
click node_partitioner "https://github.com/dpkp/kafka-python/tree/master/kafka/partitioner"
click node_consumer "https://github.com/dpkp/kafka-python/blob/master/kafka/consumer/group.py"
click node_fetcher "https://github.com/dpkp/kafka-python/blob/master/kafka/consumer/fetcher.py"
click node_subscriptions "https://github.com/dpkp/kafka-python/blob/master/kafka/consumer/subscription_state.py"
click node_coord "https://github.com/dpkp/kafka-python/blob/master/kafka/coordinator/consumer.py"
click node_assignors "https://github.com/dpkp/kafka-python/tree/master/kafka/coordinator/assignors"
click node_admincli "https://github.com/dpkp/kafka-python/blob/master/kafka/cli/admin/__init__.py"
click node_adminclient "https://github.com/dpkp/kafka-python/blob/master/kafka/admin/client.py"
click node_adminops "https://github.com/dpkp/kafka-python/tree/master/kafka/admin"
click node_cluster "https://github.com/dpkp/kafka-python/blob/master/kafka/cluster.py"
click node_network "https://github.com/dpkp/kafka-python/blob/master/kafka/net/manager.py"
click node_backend "https://github.com/dpkp/kafka-python/tree/master/kafka/net/backend"
click node_metrics "https://github.com/dpkp/kafka-python/tree/master/kafka/metrics"
click node_protocol "https://github.com/dpkp/kafka-python/tree/master/kafka/protocol"
click node_records "https://github.com/dpkp/kafka-python/blob/master/kafka/record/abc.py"
click node_codec "https://github.com/dpkp/kafka-python/blob/master/kafka/codec.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_producer,node_accum,node_sender,node_txn,node_partitioner toneBlue
class node_consumer,node_fetcher,node_subscriptions,node_coord,node_assignors,node_broker toneAmber
class node_admincli,node_adminclient,node_adminops toneMint
class node_cluster,node_network,node_backend,node_metrics toneRose
class node_protocol,node_records,node_codec,node_app,node_result toneIndigo
```

## Components

### Produce

- [Producer API](https://github.com/dpkp/kafka-python/blob/master/kafka/producer/kafka.py): `kafka/producer/kafka.py`
- [Record batching](https://github.com/dpkp/kafka-python/blob/master/kafka/producer/record_accumulator.py): `kafka/producer/record_accumulator.py`
- [Network sender](https://github.com/dpkp/kafka-python/blob/master/kafka/producer/sender.py): `kafka/producer/sender.py`
- [Transactions](https://github.com/dpkp/kafka-python/blob/master/kafka/producer/transaction_manager.py): `kafka/producer/transaction_manager.py`
- [Partitioning](https://github.com/dpkp/kafka-python/tree/master/kafka/partitioner): `kafka/partitioner`

### Consume

- [Consumer API](https://github.com/dpkp/kafka-python/blob/master/kafka/consumer/group.py): `kafka/consumer/group.py`
- [Fetching records](https://github.com/dpkp/kafka-python/blob/master/kafka/consumer/fetcher.py): `kafka/consumer/fetcher.py`
- [Subscriptions](https://github.com/dpkp/kafka-python/blob/master/kafka/consumer/subscription_state.py): `kafka/consumer/subscription_state.py`
- [Group coordination](https://github.com/dpkp/kafka-python/blob/master/kafka/coordinator/consumer.py): `kafka/coordinator/consumer.py`
- [Partition assignors](https://github.com/dpkp/kafka-python/tree/master/kafka/coordinator/assignors): `kafka/coordinator/assignors`

### Administration

- [Admin CLI](https://github.com/dpkp/kafka-python/blob/master/kafka/cli/admin/__init__.py): `kafka/cli/admin/__init__.py`
- [Admin client](https://github.com/dpkp/kafka-python/blob/master/kafka/admin/client.py): `kafka/admin/client.py`
- [Admin operations](https://github.com/dpkp/kafka-python/tree/master/kafka/admin): `kafka/admin`

### Shared client runtime

- [Cluster metadata](https://github.com/dpkp/kafka-python/blob/master/kafka/cluster.py): `kafka/cluster.py`
- [Network manager](https://github.com/dpkp/kafka-python/blob/master/kafka/net/manager.py): `kafka/net/manager.py`
- [Network backends](https://github.com/dpkp/kafka-python/tree/master/kafka/net/backend): `kafka/net/backend`
- [Metrics](https://github.com/dpkp/kafka-python/tree/master/kafka/metrics): `kafka/metrics`

### Wire and records

- [Protocol encoding](https://github.com/dpkp/kafka-python/tree/master/kafka/protocol): `kafka/protocol`
- [Record formats](https://github.com/dpkp/kafka-python/blob/master/kafka/record/abc.py): `kafka/record/abc.py`
- [Compression codecs](https://github.com/dpkp/kafka-python/blob/master/kafka/codec.py): `kafka/codec.py`

### Other components

- **Application**
- **Kafka brokers**
- **Consumer records**

## Connections

- Application → Producer API: sends records
- Producer API → Record batching: batches records (evidence: `kafka/producer/kafka.py`)
- Producer API → Partitioning: chooses partitions (evidence: `kafka/producer/kafka.py`)
- Producer API → Network sender: dispatches batches (evidence: `kafka/producer/kafka.py`)
- Producer API → Transactions: manages transactions (evidence: `kafka/producer/kafka.py`)
- Network sender → Network manager: sends requests
- Application → Consumer API: requests records
- Consumer API → Subscriptions: tracks assignments (evidence: `kafka/consumer/group.py`)
- Consumer API → Group coordination: coordinates groups (evidence: `kafka/consumer/group.py`)
- Consumer API → Fetching records: fetches records (evidence: `kafka/consumer/group.py`)
- Group coordination → Partition assignors: selects assignor (evidence: `kafka/coordinator/consumer.py`)
- Fetching records → Record formats: decodes batches (evidence: `kafka/consumer/fetcher.py`)
- Fetching records → Consumer records: returns records
- Application → Admin CLI: runs commands
- Admin CLI → Admin client: invokes client (evidence: `kafka/cli/admin/__init__.py`)
- Admin client → Admin operations: delegates operations (evidence: `kafka/admin/client.py`)
- Admin client → Network manager: sends requests (evidence: `kafka/admin/client.py`)
- Producer API → Network manager: uses networking (evidence: `kafka/producer/kafka.py`)
- Consumer API → Network manager: uses networking (evidence: `kafka/consumer/group.py`)
- Network manager → Cluster metadata: updates metadata (evidence: `kafka/net/manager.py`)
- Network manager → Network backends: runs I/O (evidence: `kafka/net/manager.py`)
- Network manager → Protocol encoding: exchanges messages (evidence: `kafka/net/manager.py`)
- Protocol encoding → Record formats: encodes record data (evidence: `kafka/protocol/old/struct.py`)
- Record formats → Compression codecs: uses compression (evidence: `kafka/record/default_records.py`)
- Producer API → Kafka brokers: produces messages
- Kafka brokers → Consumer API: provides messages
- Producer API → Metrics: exposes metrics (evidence: `kafka/producer/kafka.py`)
- Consumer API → Metrics: exposes metrics (evidence: `kafka/consumer/group.py`)
- Admin client → Kafka brokers: administers cluster
