How sa4s-serc/kafka_demo works
Project Type and Purpose
• This is a demo “kafka_demo” repository illustrating two styles of event-driven ML pipelines with Apache Kafka and Docker Compose:
a. Choreography: a simple producer → Kafka → consumer flow that applies a pre-built ML model.
b. Orchestration: a multi-step pipeline (data generation, training, model‐update, prediction) where an orchestrator service steers the flow.
Top-Level File Structure & Key Directories
• Choreography/
• producer/ (producer.py, Dockerfile)
• consumer/ (consumer.py, model.py, Dockerfile)
• docker-compose.yml (starts Zookeeper + Kafka broker)
• Orchestration/
• data_generator.py, trainer.py, orchestrator.py, predictor.py
• kd.py (Kafka topic definitions), logger_config.py
• cricket_model.pkl (trained artifact), docker-compose.yml
• README.md (usage and environment instructions)
Main Architectural Components
• Kafka Platform (Zookeeper + Broker)
• Choreography Services
• Producer Service (publishes raw events)
• Consumer Service (loads model.py, consumes events, does inference)
• Orchestration Services
• Data-Generator Service (publishes synthetic data events)
• Trainer Service (consumes data, trains model, emits model-update events)
• Orchestrator Service (controls the sequence: kicks off training, awaits completion, triggers prediction)
• Predictor Service (loads latest model artifact, consumes data, produces predictions)
Component Interactions & Data Flow
• Choreography Style: Producer → “raw‐data” topic → Consumer → (offline) prediction output
• Orchestration Style:
• Orchestrator → control topic → Trainer
• Data-Generator → “raw‐data” topic → Trainer & Predictor
• Trainer → model artifacts (cricket_model.pkl) + “model‐update” topic → Predictor
• Predictor → “prediction‐results” topic or logs
Architectural Patterns & Design Principles
• Event-driven microservices communicating via Kafka topics
• Containerized deployment via Docker Compose (each service in its own container)
• Two coordination styles:
• Choreography: each service reacts independently to events
• Orchestration: a central coordinator (Orchestrator) issues control-flow messages
• Separation of concerns: data generation, model training, inference, and control logic in distinct services
Relevant Technologies & Frameworks
• Python + kafka-python library (v2.0.2)
• Docker & Docker Compose for multi-container orchestration
• Pickle for model persistence (cricket_model.pkl)
• logger_config.py for structured logging across services
• kd.py for centralized topic‐name definitions (ensures uniform topic references)
Final Recommendations
• Both the choreography (event-driven only) and orchestration (central controller) flows to demonstrate the two architectural styles in one unified document.