# mubahsirhassan/mlops-project architecture

> Architecture diagram of the GitHub repository mubahsirhassan/mlops-project, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-21). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/mubahsirhassan/mlops-project
- Repository: https://github.com/mubahsirhassan/mlops-project

## Overview

##### Project Type and Purpose
- This is an end-to-end MLOps pipeline (maturity level 4) that covers data versioning, CI/CD, model serving, monitoring, drift detection, and continuous training.
- Core purpose: automate training, deployment, and monitoring of a ML model on AWS (ECR/EKS) with observability (Prometheus/Grafana) and drift monitoring (Streamlit).

##### Main Architectural Components
a. Source Control & Data Versioning
– GitHub repo (code, DVC metadata).
– DVC remote storage (e.g., S3) for datasets & model artifacts.
b. CI/CD Pipeline (GitHub Actions)
– main.yml workflow:
- Build Docker image
- Run pytest
- Push image to AWS ECR
- Trigger deployment to EKS.
c. Model Training & Experiment Tracking
– prediction_model module (pipeline.py, training_pipeline.py, processing/)
– MLflow tracking server (stores runs, parameters, metrics)
d. Model Serving
– FastAPI application (main.py & predict.py) packaged in Docker
– Deployed to AWS EKS (Kubernetes Deployment + Service via deployment.yml & service.yml)
– Exposes REST endpoints for real‐time and batch predictions
e. Monitoring & Observability
– FastAPI “/metrics” endpoint (Prometheus exporter)
– Prometheus scrapes metrics
f. Drift Monitoring App
– Streamlit app (drift_monitoring/app_v1.py)
– Connects to production data stream and historical data in DVC to detect data/target drift and quality issues

##### Relationships & Data Flows
- Data flow from raw data → DVC → preprocessing → training → trained model → DVC → MLflow
- CI/CD flow: Git push → GitHub Actions → Docker/ECR → Kubernetes
- Request flow: Client → FastAPI (EKS) → Model artifact
- Monitoring flow: FastAPI → Prometheus → Grafana
- Drift flow: Production data store → Streamlit app → Visual reports

##### Architectural Patterns & Principles
- Microservices (FastAPI service separated from drift App)
- GitOps/CICD (GitHub Actions triggers)
- Infrastructure as code (Kubernetes YAMLs, Dockerfile)
- DataOps/Data Versioning (DVC)
- Observability (Prometheus/Grafana)
- Experiment tracking (MLflow)

##### Technologies & Frameworks
- DVC (data + model versioning)
- GitHub Actions (CI/CD)
- Docker & AWS ECR (container registry)
- AWS EKS (Kubernetes cluster)
- FastAPI (model serving)
- MLflow (experiment tracking)
- Prometheus (metrics collection)
- Grafana (visualization)
- Streamlit (drift monitoring UI)
- PyTest (unit tests)

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Source Control & Data Versioning
    subgraph "DataOps Zone"
        direction TB
        GH["GitHub Repo (code & DVC metadata)"]:::orange
        dvc_config[".dvc/config"]:::green
        dvc_ignore[".dvcignore"]:::green
        datasets_dvc["prediction_model/datasets.dvc"]:::green
        trained_models_dvc["prediction_model/trained_models.dvc"]:::green
        raw_data["prediction_model/datasets/"]:::green
        DVC_remote["DVC Remote Storage (S3)"]:::green
    end

    %% CI/CD Pipeline
    subgraph "CI/CD Zone"
        direction TB
        Actions["GitHub Actions"]:::orange
        main_yml[".github/workflows/main.yml"]:::orange
        Dockerfile["Dockerfile"]:::orange
        ECR["AWS ECR"]:::blue
    end

    %% Model Training & Experiment Tracking
    subgraph "Training & Tracking Zone"
        direction TB
        pipeline_py["prediction_model/pipeline.py"]:::blue
        preprocessing_py["prediction_model/processing/preprocessing.py"]:::blue
        data_handling_py["prediction_model/processing/data_handling.py"]:::blue
        training_pipeline_py["prediction_model/training_pipeline.py"]:::blue
        MLflow["MLflow Tracking Server"]:::purple
    end

    %% Model Serving
    subgraph "Serving Zone"
        direction TB
        EKS["AWS EKS Cluster"]:::blue
        dep_yml["deployment.yml"]:::blue
        svc_yml["service.yml"]:::blue
        FastAPI["FastAPI Service"]:::blue
        main_py["main.py"]:::blue
        predict_py["prediction_model/predict.py"]:::blue
    end

    %% Monitoring & Observability
    subgraph "Monitoring Zone"
        direction TB
        Prometheus["Prometheus"]:::purple
        Grafana["Grafana"]:::purple
    end

    %% Drift Monitoring App
    subgraph "Drift Monitoring Zone"
        direction TB
        Streamlit["Streamlit App"]:::purple
        drift_app["drift_monitoring/app_v1.py"]:::purple
    end

    %% Testing
    tests["tests/test_prediction.py"]:::green

    %% Connections: DataOps
    raw_data -->|"dvc push/pull"| DVC_remote
    datasets_dvc -->|"metadata"| DVC_remote
    trained_models_dvc -->|"metadata"| DVC_remote
    GH -->|push code & metadata| Actions

    %% CI/CD Flow
    Actions -->|"workflow"| main_yml
    Actions -->|"build image"| Dockerfile
    Actions -->|"push image"| ECR
    ECR -->|container image| EKS
    EKS -->|deploy via YAMLs| dep_yml
    EKS -->|service| svc_yml
    svc_yml -->|routes to| FastAPI

    %% Training & Tracking Flow
    raw_data -->|"ingest"| pipeline_py
    pipeline_py -->|"preprocess"| preprocessing_py
    pipeline_py -->|"handle data"| data_handling_py
    pipeline_py -->|"start training"| training_pipeline_py
    training_pipeline_py -->|"log runs"| MLflow
    training_pipeline_py -->|"store models"| trained_models_dvc

    %% Serving Flow
    FastAPI -->|calls| predict_py
    FastAPI -->|exposes metrics| Prometheus

    %% Monitoring Flow
    Prometheus -->|"scrape metrics"| Grafana

    %% Drift Flow
    raw_data -->|live data| Streamlit
    DVC_remote -->|historical data| Streamlit
    Streamlit -->|renders UI| drift_app

    %% Testing Flow
    tests -->|tests pipeline| pipeline_py
    tests -->|tests endpoint| predict_py

    %% Click Events
    click dvc_config "https://github.com/mubahsirhassan/mlops-project/tree/main/.dvc/config"
    click dvc_ignore "https://github.com/mubahsirhassan/mlops-project/blob/main/.dvcignore"
    click datasets_dvc "https://github.com/mubahsirhassan/mlops-project/blob/main/prediction_model/datasets.dvc"
    click trained_models_dvc "https://github.com/mubahsirhassan/mlops-project/blob/main/prediction_model/trained_models.dvc"
    click raw_data "https://github.com/mubahsirhassan/mlops-project/tree/main/prediction_model/datasets/"
    click main_yml "https://github.com/mubahsirhassan/mlops-project/blob/main/.github/workflows/main.yml"
    click Dockerfile "https://github.com/mubahsirhassan/mlops-project/tree/main/Dockerfile"
    click pipeline_py "https://github.com/mubahsirhassan/mlops-project/blob/main/prediction_model/pipeline.py"
    click preprocessing_py "https://github.com/mubahsirhassan/mlops-project/blob/main/prediction_model/processing/preprocessing.py"
    click data_handling_py "https://github.com/mubahsirhassan/mlops-project/blob/main/prediction_model/processing/data_handling.py"
    click training_pipeline_py "https://github.com/mubahsirhassan/mlops-project/blob/main/prediction_model/training_pipeline.py"
    click main_py "https://github.com/mubahsirhassan/mlops-project/blob/main/main.py"
    click predict_py "https://github.com/mubahsirhassan/mlops-project/blob/main/prediction_model/predict.py"
    click drift_app "https://github.com/mubahsirhassan/mlops-project/blob/main/drift_monitoring/app_v1.py"
    click tests "https://github.com/mubahsirhassan/mlops-project/blob/main/tests/test_prediction.py"

    %% Styles
    classDef orange fill:#FFA500,stroke:#333,stroke-width:1px
    classDef green fill:#8FBC8F,stroke:#333,stroke-width:1px
    classDef blue fill:#1E90FF,stroke:#333,stroke-width:1px
    classDef purple fill:#9370DB,stroke:#333,stroke-width:1px
```
