How blizzard/node-rdkafka works
Purpose and Entry Points
node-rdkafka is a high-performance Node.js binding for Apache Kafka, backed by native librdkafka 2.12.0. Consumers import the package through librdkafka.js, which loads the compiled addon and exposes the JavaScript API assembled in lib/index.js.
• librdkafka.js
• lib/index.js
• src/binding.cc
JavaScript Client Layer
The public JS layer provides EventEmitter-oriented client abstractions, validates/normalizes inputs, and maps native results into Node callbacks and events. A shared base client owns connection lifecycle, metadata queries, and common event behavior.
• lib/client.js
• lib/producer.js
• lib/kafka-consumer.js
Producer Pipeline
Producer accepts messages, topic/partition/key/timestamp data, and dispatches them to the native producer. Delivery reports, errors, polling, flushes, and transactional operations return asynchronously through the event/callback boundary. HighLevelProducer adds per-message delivery callbacks and serializers; writable stream support wraps the standard producer.
• lib/producer.js
• lib/producer/high-level-producer.js
• src/producer.cc
Consumer Pipeline
KafkaConsumer connects to Kafka consumer groups, manages subscriptions and offsets, and supports flowing or bounded consumption. Native worker activity feeds message batches/events back to JS; the JS facade emits data, rebalance, commit, EOF, warning, and broker events. A readable-stream adapter provides the simpler stream API.
• lib/kafka-consumer.js
• lib/kafka-consumer-stream.js
• src/kafka-consumer.cc
Native Addon and Callback Boundary
C++ code wraps librdkafka handles and translates Node/V8 values, configuration, errors, topic-partition objects, and librdkafka callbacks. Connection and callback machinery form the key concurrency boundary: Kafka/network work occurs in librdkafka/native workers, while results are marshaled onto Node-visible callbacks/events.
• src/binding.cc
• src/connection.cc
• src/callbacks.cc
Administrative and Topic Operations
The admin client exposes asynchronous topic creation, deletion, and partition expansion. Topic and topic-partition representations are shared transport types for produce, consume, offset commit, metadata, and admin operations.
• lib/admin.js
• src/admin.cc
• src/topic.cc
Runtime, Dependencies, and Build Boundary
The runtime requires Node.js >=16, OpenSSL, and Kafka >=0.9. The native addon is built through node-gyp/GYP and links the vendored librdkafka submodule on Linux/macOS. Windows instead downloads and links a version-matched NuGet librdkafka.redist binary. Kafka brokers are the primary external service; local integration environments are supported through Docker Compose.
• binding.gyp
• deps/librdkafka.gyp
• docker-compose.yml
Components 19
JavaScript API
- Package entry · Loads the compiled addon and public API.
librdkafka.js - Public API exports
lib/index.js - Base client · Owns connection lifecycle, metadata, and common events.
lib/client.js - Producer
lib/producer.js - High-level producer · Adds serializers and per-message delivery callbacks.
lib/producer/high-level-producer.js - Kafka consumer · Emits messages, rebalance, commit, EOF, warning, and broker events.
lib/kafka-consumer.js - Consumer stream
lib/kafka-consumer-stream.js - Admin client · Creates, deletes, and expands topics asynchronously.
lib/admin.js
Native Addon
- Addon binding · Exposes C++ wrappers to JavaScript.
src/binding.cc - Connection layer · Wraps librdkafka handles and connection operations.
src/connection.cc - Callback bridge · Marshals librdkafka worker results into Node callbacks and events.
src/callbacks.cc - Native producer
src/producer.cc - Native consumer · Feeds native message batches and events to JavaScript.
src/kafka-consumer.cc - Native admin
src/admin.cc - Topic transport types · Shared topic and topic-partition objects.
src/topic.cc
Build & Runtime
- node-gyp build · Builds the addon and links platform dependencies.
binding.gyp - Vendored librdkafka · librdkafka 2.12.0 is linked on Linux and macOS.
deps/librdkafka.gyp - Local integration environment
docker-compose.yml
Other components
- Kafka brokers · Primary broker and consumer-group service.
Connections 26
- Package entry → Public API exports: loads
- Package entry → Addon binding: loads addon
- Public API exports → Base client: exports
- Public API exports → Producer: exports
- Public API exports → Kafka consumer: exports
- Public API exports → Admin client: exports
- Producer → Base client: extends
- High-level producer → Producer: extends
- Kafka consumer → Base client: extends
- Consumer stream → Kafka consumer: wraps
- Producer → Native producer: produce and transactions
- Kafka consumer → Native consumer: subscribe and consume
- Admin client → Native admin: topic operations
- Addon binding → Connection layer: initializes
- Addon binding → Native producer: exposes
- Addon binding → Native consumer: exposes
- Addon binding → Native admin: exposes
- Connection layer → Callback bridge: routes native results
- Callback bridge → Base client: Node callbacks and events
- Native producer → Topic transport types: uses
- Native consumer → Topic transport types: uses
- Native admin → Topic transport types: uses
- Connection layer → Kafka brokers: connects through librdkafka
- node-gyp build → Addon binding: builds
- node-gyp build → Vendored librdkafka: links
- Local integration environment → Kafka brokers: provides local environment