How dpkp/kafka-python works
kafka-python is a Python Apache Kafka client. Its principal workflow is applications producing serialized records to Kafka and consumers fetching, decoding, and returning records; grouped consumers additionally coordinate partition assignment and offset commits. The library also provides cluster administration through a client and CLI. Shared networking, metadata, protocol, record, metrics, and error components support these workflows. README statements establish the product capabilities; source-index references support several internal links, while unsampled modules are represented at their existing paths without claiming their internals were inspected.
Main entry points: kafka.py Producer API · group.py Consumer API · manager.py Network manager · client.py Admin client
Components 23
Produce
- Producer API
kafka/producer/kafka.py - Record batching
kafka/producer/record_accumulator.py - Network sender
kafka/producer/sender.py - Transactions
kafka/producer/transaction_manager.py - Partitioning
kafka/partitioner
Consume
- Consumer API
kafka/consumer/group.py - Fetching records
kafka/consumer/fetcher.py - Subscriptions
kafka/consumer/subscription_state.py - Group coordination
kafka/coordinator/consumer.py - Partition assignors
kafka/coordinator/assignors
Administration
- Admin CLI
kafka/cli/admin/__init__.py - Admin client
kafka/admin/client.py - Admin operations
kafka/admin
Shared client runtime
- Cluster metadata
kafka/cluster.py - Network manager
kafka/net/manager.py - Network backends
kafka/net/backend - Metrics
kafka/metrics
Wire and records
- Protocol encoding
kafka/protocol - Record formats
kafka/record/abc.py - Compression codecs
kafka/codec.py
Other components
- Application
- Kafka brokers
- Consumer records
Connections 29
- Application → Producer API: sends records
- Producer API → Record batching: batches records
kafka/producer/kafka.py - Producer API → Partitioning: chooses partitions
kafka/producer/kafka.py - Producer API → Network sender: dispatches batches
kafka/producer/kafka.py - Producer API → Transactions: manages transactions
kafka/producer/kafka.py - Network sender → Network manager: sends requests
- Application → Consumer API: requests records
- Consumer API → Subscriptions: tracks assignments
kafka/consumer/group.py - Consumer API → Group coordination: coordinates groups
kafka/consumer/group.py - Consumer API → Fetching records: fetches records
kafka/consumer/group.py - Group coordination → Partition assignors: selects assignor
kafka/coordinator/consumer.py - Fetching records → Record formats: decodes batches
kafka/consumer/fetcher.py - Fetching records → Consumer records: returns records
- Application → Admin CLI: runs commands
- Admin CLI → Admin client: invokes client
kafka/cli/admin/__init__.py - Admin client → Admin operations: delegates operations
kafka/admin/client.py - Admin client → Network manager: sends requests
kafka/admin/client.py - Producer API → Network manager: uses networking
kafka/producer/kafka.py - Consumer API → Network manager: uses networking
kafka/consumer/group.py - Network manager → Cluster metadata: updates metadata
kafka/net/manager.py - Network manager → Network backends: runs I/O
kafka/net/manager.py - Network manager → Protocol encoding: exchanges messages
kafka/net/manager.py - Protocol encoding → Record formats: encodes record data
kafka/protocol/old/struct.py - Record formats → Compression codecs: uses compression
kafka/record/default_records.py - Producer API → Kafka brokers: produces messages
- Kafka brokers → Consumer API: provides messages
- Producer API → Metrics: exposes metrics
kafka/producer/kafka.py - Consumer API → Metrics: exposes metrics
kafka/consumer/group.py - Admin client → Kafka brokers: administers cluster