# deaneeth/telco-churn-mlops-pipeline architecture

> Architecture diagram of the GitHub repository deaneeth/telco-churn-mlops-pipeline, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-10-12). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/deaneeth/telco-churn-mlops-pipeline
- Repository: https://github.com/deaneeth/telco-churn-mlops-pipeline

## Overview

##### Project Type and Purpose
- This is not just a typical web app but a production-grade MLOps pipeline for telecom customer churn prediction.
- It implements end-to-end automation: batch and streaming data ingestion (CSV + Kafka), feature engineering, model training (scikit-learn & PySpark), experiment tracking (MLflow), orchestration (Airflow), real-time inference (Kafka consumer + Flask REST API), and monitoring/logging.

##### File Structure
- Top-level modules:
- src/ – core Python packages: data, models, streaming, inference, api, utils
- pipelines/ – standalone ML workflows (sklearn & Spark)
- dags/ + airflow_home/ – Airflow DAG definitions and home dir (orchestration)
- scripts/ – helper scripts for Kafka topic creation, demos, testing
- data/ – raw & processed datasets
- artifacts/ – saved models, metrics, logs, predictions
- mlruns/ – MLflow experiment tracking storage
- tests/ + notebooks/ – unit/integration tests and exploratory notebooks
- External dependencies and config:
- Dockerfile + docker-compose.yml(.kafka)
- config.py/.yaml, Makefile, setup.py, GitHub workflows

##### Key Architectural Patterns and Components
- Layered pipeline pattern: Ingestion → Preprocess → Train → Register → Serve
- Event-driven microservices via Kafka: producer (batch/stream) → topics → consumer
- Orchestration via Airflow DAGs that coordinate batch, streaming, retraining
- Experiment management via MLflow model registry
- Containerized deployment: Flask API inside Docker, optional Gunicorn
- Monitoring/logging: logs in artifacts/logs, Airflow UI, MLflow UI

##### Main Components
A. Data Ingestion
- Static CSV loader (src/data/load_data.py)
- Kafka Producer (src/streaming/producer.py) – batch & streaming modes
B. Data Processing & Modeling
- Preprocessing module (src/data/preprocess.py)
- Feature store / processed data (data/processed/)
- Model training – scikit-learn (src/models/train.py, pipelines/sklearn_pipeline.py)
- Distributed training – PySpark (pipelines/spark_pipeline.py)
- MLflow Tracking Server & Model Registry (mlruns/)
C. Message Bus
- Kafka Cluster (docker-compose.kafka.yml)
- Topics: telco.raw.customers, telco.predictions.churn, dead-letter queue
D. Real-Time Inference
- Kafka Consumer (src/streaming/consumer.py)
- Inference backend (src/streaming/inference_backend.py)
- Schema validation (src/streaming/schema_validator.py)
E. Batch Inference & REST API
- Batch Predict (src/inference/batch_predict.py)
- Flask API with endpoints /ping and /predict (src/api/app.py)
F. Orchestration & Scheduling
- Airflow DAGs (dags/telco_churn_dag.py, kafka_batch_dag.py, kafka_streaming_dag.py)
- Airflow Scheduler & Webserver (airflow_home/)
G. Monitoring & Logging
- Logging util (src/utils/logger.py)
- Artifacts: logs, metrics JSON, reports/
- MLflow UI, Airflow UI, Kafka metrics

##### Relationships & Data Flows
1. CSV → Validation → Preprocessing → (a) Write processed arrays → (b) Trigger model training
2. Training → Model artifact → MLflow registry
3. Producer (batch/stream) reads CSV or sample → publishes JSON events → Kafka raw topic
4. Consumer subscribes to raw topic → loads model from MLflow or local artifact → predicts → publishes to predictions topic
5. REST API may call the same inference code to serve on-demand predictions
6. Airflow DAGs schedule and chain steps 1–4, handle retries, and aggregate reports
- Feedback loops: metrics and artifacts feed back into MLflow and reports

##### Architectural Patterns & Design Principles
- Separation of concerns: isolation of data ingestion, feature engineering, model training, serving, orchestration
- Event-driven architecture: Kafka for decoupling producers/consumers
- Micro-pipelines: modular Python packages, pipelines driven as scripts or via DAGs
- Idempotency & checkpointing: producer checkpoint.json, dead-letter queue for failures
- Experiment reproducibility: MLflow, consistent config via config.py/yaml, Docker images

##### Technologies & Frameworks
- Python 3.13, scikit-learn, PySpark, pandas, NumPy
- MLflow for experiment tracking & model registry
- Apache Kafka (kafka-python, Redpanda)
- Apache Airflow for orchestration
- Flask (Gunicorn) for REST API
- Docker & docker-compose for containerization
- pytest for testing, schema validation via JSON Schemas

##### Detail Without Over-Thinking
- MLflow registry as a distinct component
- Interplay between batch and streaming pipelines
- Where artifacts (models, metrics, logs) are written and read

## Diagram (Mermaid)

```mermaid
graph TB
    %% Ingestion Layer
    subgraph "Ingestion Layer"
        direction TB
        CSVLoader["Static CSV Loader\n(src/data/load_data.py)"]:::ing
        KafkaProducer["Kafka Producer\n(src/streaming/producer.py)"]:::ing
    end

    %% Processing & Training Layer
    subgraph "Processing & Training"
        direction TB
        Preprocess["Preprocessing Module\n(src/data/preprocess.py)"]:::proc
        FeatureStore[(Feature Store\ndata/processed/)]:::storage
        SKlearnTrain["Scikit-learn Training\n(src/models/train.py)"]:::proc
        SKlearnPipeline["Sklearn Pipeline\n(pipelines/sklearn_pipeline.py)"]:::proc
        SparkPipeline["PySpark Pipeline\n(pipelines/spark_pipeline.py)"]:::proc
        MLflowStore[(MLflow Store\nmlruns/)]:::storage
    end

    %% Messaging & Storage
    subgraph "Messaging & Storage"
        direction TB
        KafkaCluster[(Kafka Cluster)]:::messaging
        RawTopic[(telco.raw.customers)]:::messaging
        PredTopic[(telco.predictions.churn)]:::messaging
        DeadLetter[(Dead-letter Queue)]:::messaging
        KafkaScripts["Topic Setup Scripts\n(scripts/kafka_create_topics.sh/.ps1)"]:::messaging
        Schemas["JSON Schemas\n(schemas/*.json)"]:::messaging
    end

    %% Real-Time Inference Layer
    subgraph "Real-Time Inference"
        direction TB
        SchemaVal["Schema Validator\n(src/streaming/schema_validator.py)"]:::serv
        InferenceBackend["Inference Backend\n(src/streaming/inference_backend.py)"]:::serv
        KafkaConsumer["Kafka Consumer\n(src/streaming/consumer.py)"]:::serv
    end

    %% Serving & APIs Layer
    subgraph "Serving & APIs"
        direction TB
        BatchPredict["Batch Prediction\n(src/inference/batch_predict.py)"]:::serv
        FlaskAPI["Flask API\n(src/api/app.py)"]:::serv
    end

    %% Orchestration Layer
    subgraph "Orchestration & Scheduling"
        direction TB
        AirflowDAG["Airflow DAGs\n(dags/telco_churn_dag.py)"]:::orch
        AirflowHome[(Airflow Home\nairflow_home/)]:::orch
    end

    %% Monitoring & Logging
    subgraph "Monitoring & Logging"
        direction TB
        Logger["Logging Util\n(src/utils/logger.py)"]:::misc
        LogArtifacts[(Logs\nartifacts/logs/)]:::misc
        MetricsArtifacts[(Metrics\nartifacts/metrics/)]:::misc
        Reports[(Reports\nartifacts/reports/)]:::misc
    end

    %% Infrastructure
    subgraph "Infrastructure"
        direction TB
        Config["Config File\n(config.py)"]:::infra
        DockerfileNode["Dockerfile"]:::infra
        Compose["docker-compose.yml"]:::infra
    end

    %% Data Flow & Relationships
    CSVLoader -->|reads CSV| Preprocess
    Preprocess -->|outputs arrays| FeatureStore
    Preprocess -->|triggers training| SKlearnTrain
    SKlearnTrain -->|uses pipeline| SKlearnPipeline
    SKlearnTrain -->|registers model| MLflowStore
    SparkPipeline -->|registers model| MLflowStore

    KafkaProducer -->|publishes JSON| KafkaCluster
    KafkaCluster --> RawTopic
    RawTopic -->|stream of JSON| KafkaConsumer

    KafkaConsumer -->|validate| SchemaVal
    KafkaConsumer -->|predict| InferenceBackend
    InferenceBackend --> MLflowStore
    InferenceBackend -->|publishes results| PredTopic
    PredTopic --> DeadLetter

    BatchPredict -->|loads model| MLflowStore
    BatchPredict --> LogArtifacts

    FlaskAPI -->|HTTP /predict| MLflowStore

    AirflowDAG --> CSVLoader
    AirflowDAG --> KafkaProducer
    AirflowDAG --> Preprocess
    AirflowDAG --> SKlearnTrain
    AirflowDAG --> SparkPipeline
    AirflowDAG --> BatchPredict
    AirflowDAG --> AirflowHome

    Logger --> LogArtifacts
    Logger --> MetricsArtifacts
    Logger --> Reports

    Config --> CSVLoader
    Config --> KafkaProducer
    Config --> Preprocess
    Config --> SKlearnTrain
    Config --> SparkAPI

    Compose --> KafkaCluster
    Compose --> AirflowHome
    Compose --> FlaskAPI

    %% Click Events
    click CSVLoader "https://github.com/deaneeth/telco-churn-mlops-pipeline/blob/main/src/data/load_data.py"
    click KafkaProducer "https://github.com/deaneeth/telco-churn-mlops-pipeline/blob/main/src/streaming/producer.py"
    click Preprocess "https://github.com/deaneeth/telco-churn-mlops-pipeline/blob/main/src/data/preprocess.py"
    click FeatureStore "https://github.com/deaneeth/telco-churn-mlops-pipeline/tree/main/data/processed/"
    click SKlearnTrain "https://github.com/deaneeth/telco-churn-mlops-pipeline/blob/main/src/models/train.py"
    click SKlearnPipeline "https://github.com/deaneeth/telco-churn-mlops-pipeline/blob/main/pipelines/sklearn_pipeline.py"
    click SparkPipeline "https://github.com/deaneeth/telco-churn-mlops-pipeline/blob/main/pipelines/spark_pipeline.py"
    click MLflowStore "https://github.com/deaneeth/telco-churn-mlops-pipeline/tree/main/mlruns/"
    click KafkaScripts "https://github.com/deaneeth/telco-churn-mlops-pipeline/blob/main/scripts/kafka_create_topics.sh"
    click KafkaScripts "https://github.com/deaneeth/telco-churn-mlops-pipeline/blob/main/scripts/kafka_create_topics.ps1"
    click Schemas "https://github.com/deaneeth/telco-churn-mlops-pipeline/blob/main/schemas/telco_customer_schema.json"
    click Schemas "https://github.com/deaneeth/telco-churn-mlops-pipeline/blob/main/schemas/churn_prediction_schema.json"
    click Schemas "https://github.com/deaneeth/telco-churn-mlops-pipeline/blob/main/schemas/deadletter_schema.json"
    click KafkaConsumer "https://github.com/deaneeth/telco-churn-mlops-pipeline/blob/main/src/streaming/consumer.py"
    click InferenceBackend "https://github.com/deaneeth/telco-churn-mlops-pipeline/blob/main/src/streaming/inference_backend.py"
    click SchemaVal "https://github.com/deaneeth/telco-churn-mlops-pipeline/blob/main/src/streaming/schema_validator.py"
    click BatchPredict "https://github.com/deaneeth/telco-churn-mlops-pipeline/blob/main/src/inference/batch_predict.py"
    click FlaskAPI "https://github.com/deaneeth/telco-churn-mlops-pipeline/blob/main/src/api/app.py"
    click AirflowDAG "https://github.com/deaneeth/telco-churn-mlops-pipeline/blob/main/dags/telco_churn_dag.py"
    click AirflowHome "https://github.com/deaneeth/telco-churn-mlops-pipeline/tree/main/airflow_home/"
    click Logger "https://github.com/deaneeth/telco-churn-mlops-pipeline/blob/main/src/utils/logger.py"
    click LogArtifacts "https://github.com/deaneeth/telco-churn-mlops-pipeline/tree/main/artifacts/logs/"
    click MetricsArtifacts "https://github.com/deaneeth/telco-churn-mlops-pipeline/tree/main/artifacts/metrics/"
    click Reports "https://github.com/deaneeth/telco-churn-mlops-pipeline/tree/main/artifacts/reports/"
    click Config "https://github.com/deaneeth/telco-churn-mlops-pipeline/blob/main/config.py"
    click DockerfileNode "https://github.com/deaneeth/telco-churn-mlops-pipeline/tree/main/Dockerfile"
    click Compose "https://github.com/deaneeth/telco-churn-mlops-pipeline/blob/main/docker-compose.yml"

    %% Styles
    classDef ing fill:#e6ffe6,stroke:#00b300,color:#006600
    classDef proc fill:#e6f0ff,stroke:#0066cc,color:#003366
    classDef serv fill:#fff0e6,stroke:#ff6600,color:#994d00
    classDef orch fill:#f2e6ff,stroke:#9933ff,color:#660099
    classDef messaging fill:#f2f2f2,stroke:#999999,color:#333333
    classDef storage fill:#f2f2f2,stroke:#999999,color:#333333
    classDef misc fill:#ffffe6,stroke:#cccc00,color:#666600
    classDef infra fill:#e6e6e6,stroke:#666666,color:#333333
```
