How bngiahuy/kafka-master works
Based on our analysis of the provided file tree and README, we’re looking at a distributed job-processing system—the “Kafka Master”—that serves as the coordinator (Master Node) for assigning work to various Worker Nodes. Communication across the system is handled via Kafka for asynchronous messaging, while Redis is used to keep track of worker statuses, job assignments, and system state. The project appears to be implemented as a Node.js application (located in the src/ directory) for the master services and exposes API endpoints to report system status and update configurations. There is also a dedicated Python component (in the python/ folder) that likely implements the workers (or a specific type of worker), each with it’s own Dockerfile and docker-compose configuration for containerized deployment.
Main Components
• Master Node: This is the core of the application. It hosts the main controller logic (src/index.js), configuration (inside src/configs/), API endpoints (routes and controllers), producer and consumer modules (src/producers & src/consumers), and additional utilities (src/utils and leaderElection).
• Worker Nodes: Represented by the Python-based code (python/worker.py) that likely runs in separate containers. They register with the master, listen for job assignments, and report back progress.
• Kafka: Acts as the messaging backbone with multiple topics (e.g., KAFKA_TOPIC_NAME_MASTER, KAFKA_TOPIC_NAME_WORKER, KAFKA_TOPIC_NAME_WORKER_FREE) used for distributing tasks and receiving updates.
• Redis: Used as a persistent and fast-access data store to track the system’s runtime state (worker statuses, job batches, leader election metadata, and file/chunk processing information).
Relationships and Interactions
• The Master Node reads input files and splits them into data batches. It then publishes these job assignments to the Kafka topic (master signal).
• Worker Nodes subscribe to a specific Kafka topic channel to pick up jobs. Upon receiving a job, they process the data and communicate their status (or completion) back through another Kafka topic.
• The Master continuously monitors worker progress using both Kafka messages and Redis, where it tracks registration, status updates, and job/chunk states.
• API endpoints exposed by the Master Node (documented in the README) allow external monitoring and adjustments (like updating batch sizes or Kafka partition information).
Architectural Patterns and Design Principles
• Service-Oriented/Modular Design: Notice the separation of components into logically distinct directories such as controllers, services, producers, and consumers.
• Event-Driven Architecture: The usage of Kafka topics clearly indicates an event-driven communication pattern, emphasizing decoupled and asynchronous processing.
• Containerization and Microservices: Docker and docker-compose files at both the project root and within the python subdirectory suggest that each component is containerized and potentially deployed as distinct services (for scaling or isolated development).
Incorporate Technologies, Frameworks, and Libraries
• Node.js/Express for the Master API and orchestration logic.
• Kafka for message brokering with configured topics.
• Redis as a fast key-value store for real-time state and status.
• Docker for containerization and easy deployment, including separate setups for Node.js and Python worker environments.
• The file structure also includes utility modules (logging, helper functions) that support the system’s maintenance and reliability.
Final Touches
• Components are clearly labeled.
• From Master → Kafka → Worker (job assignments).
• From Worker → Kafka → Master (heartbeat & processing updates).
• From Master → Redis and vice versa (status tracking, job chunk management).
• Differentiate between communication protocols and data flows where applicable (e.g., API calls vs. message queues).