How deaneeth/telco-churn-mlops-pipeline works
Project Type and Purpose
• This is not just a typical web app but a production-grade MLOps pipeline for telecom customer churn prediction.
• It implements end-to-end automation: batch and streaming data ingestion (CSV + Kafka), feature engineering, model training (scikit-learn & PySpark), experiment tracking (MLflow), orchestration (Airflow), real-time inference (Kafka consumer + Flask REST API), and monitoring/logging.
File Structure
• Top-level modules:
• src/ – core Python packages: data, models, streaming, inference, api, utils
• pipelines/ – standalone ML workflows (sklearn & Spark)
• dags/ + airflow_home/ – Airflow DAG definitions and home dir (orchestration)
• scripts/ – helper scripts for Kafka topic creation, demos, testing
• data/ – raw & processed datasets
• artifacts/ – saved models, metrics, logs, predictions
• mlruns/ – MLflow experiment tracking storage
• tests/ + notebooks/ – unit/integration tests and exploratory notebooks
• External dependencies and config:
• Dockerfile + docker-compose.yml(.kafka)
• config.py/.yaml, Makefile, setup.py, GitHub workflows
Key Architectural Patterns and Components
• Layered pipeline pattern: Ingestion → Preprocess → Train → Register → Serve
• Event-driven microservices via Kafka: producer (batch/stream) → topics → consumer
• Orchestration via Airflow DAGs that coordinate batch, streaming, retraining
• Experiment management via MLflow model registry
• Containerized deployment: Flask API inside Docker, optional Gunicorn
• Monitoring/logging: logs in artifacts/logs, Airflow UI, MLflow UI
Main Components
A. Data Ingestion
• Static CSV loader (src/data/load_data.py)
• Kafka Producer (src/streaming/producer.py) – batch & streaming modes
B. Data Processing & Modeling
• Preprocessing module (src/data/preprocess.py)
• Feature store / processed data (data/processed/)
• Model training – scikit-learn (src/models/train.py, pipelines/sklearn_pipeline.py)
• Distributed training – PySpark (pipelines/spark_pipeline.py)
• MLflow Tracking Server & Model Registry (mlruns/)
C. Message Bus
• Kafka Cluster (docker-compose.kafka.yml)
• Topics: telco.raw.customers, telco.predictions.churn, dead-letter queue
D. Real-Time Inference
• Kafka Consumer (src/streaming/consumer.py)
• Inference backend (src/streaming/inference_backend.py)
• Schema validation (src/streaming/schema_validator.py)
E. Batch Inference & REST API
• Batch Predict (src/inference/batch_predict.py)
• Flask API with endpoints /ping and /predict (src/api/app.py)
F. Orchestration & Scheduling
• Airflow DAGs (dags/telco_churn_dag.py, kafka_batch_dag.py, kafka_streaming_dag.py)
• Airflow Scheduler & Webserver (airflow_home/)
G. Monitoring & Logging
• Logging util (src/utils/logger.py)
• Artifacts: logs, metrics JSON, reports/
• MLflow UI, Airflow UI, Kafka metrics
Relationships & Data Flows
1. CSV → Validation → Preprocessing → (a) Write processed arrays → (b) Trigger model training
2. Training → Model artifact → MLflow registry
3. Producer (batch/stream) reads CSV or sample → publishes JSON events → Kafka raw topic
4. Consumer subscribes to raw topic → loads model from MLflow or local artifact → predicts → publishes to predictions topic
5. REST API may call the same inference code to serve on-demand predictions
6. Airflow DAGs schedule and chain steps 1–4, handle retries, and aggregate reports
• Feedback loops: metrics and artifacts feed back into MLflow and reports
Architectural Patterns & Design Principles
• Separation of concerns: isolation of data ingestion, feature engineering, model training, serving, orchestration
• Event-driven architecture: Kafka for decoupling producers/consumers
• Micro-pipelines: modular Python packages, pipelines driven as scripts or via DAGs
• Idempotency & checkpointing: producer checkpoint.json, dead-letter queue for failures
• Experiment reproducibility: MLflow, consistent config via config.py/yaml, Docker images
Technologies & Frameworks
• Python 3.13, scikit-learn, PySpark, pandas, NumPy
• MLflow for experiment tracking & model registry
• Apache Kafka (kafka-python, Redpanda)
• Apache Airflow for orchestration
• Flask (Gunicorn) for REST API
• Docker & docker-compose for containerization
• pytest for testing, schema validation via JSON Schemas
Detail Without Over-Thinking
• MLflow registry as a distinct component
• Interplay between batch and streaming pipelines
• Where artifacts (models, metrics, logs) are written and read