# mohammmedab/mlops-pipeline-bike-trip-duration architecture

> Architecture diagram of the GitHub repository mohammmedab/mlops-pipeline-bike-trip-duration, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-01-05). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/mohammmedab/mlops-pipeline-bike-trip-duration
- Repository: https://github.com/mohammmedab/mlops-pipeline-bike-trip-duration

## Overview

##### Main Components

a) Data Layer:
- AWS S3 Bucket (Training Data Storage)
- PostgreSQL Database (MLflow Backend & Monitoring)
- Data flow paths for both training and prediction data

b) Training & Experimentation Layer:
- MLflow Server
- Experiment Tracking
- Model Registry
- Prefect Orchestration
- Training Flow
- Batch Metrics Calculation Flow

c) Serving Layer:
- Flask API
- Docker Container
- Gunicorn Server

d) Monitoring Layer:
- Evidently (Metrics Calculation)
- Grafana Dashboards
- PostgreSQL Monitoring Database

##### Component Relationships

- S3 → Prefect (Training data flow)
- Prefect → MLflow (Model training and registration)
- MLflow → Flask API (Model serving)
- Flask API → PostgreSQL (Prediction logging)
- PostgreSQL → Evidently (Metrics calculation)
- Evidently → Monitoring DB → Grafana (Monitoring flow)

##### Infrastructure Layer

AWS services as the underlying infrastructure:
- EC2 Instance (hosting MLflow, Grafana, Monitoring DB)
- RDS PostgreSQL
- S3 Bucket

##### Specific Design Elements

Use the following visual elements:
- Blue: AWS Services
- Green: Data Storage
- Orange: Processing Services
- Purple: Monitoring Components

- Rectangles: Services

##### Important Details

- Prefect flows (orchestration/batch_metrics_calculation.py and orchestration/train.py)
- Web service endpoint (web/predict.py)
- Database connections (database/sql.py)
- Monitoring components (monitoring/dummy_metrics.py)

## Diagram (Mermaid)

```mermaid
graph TB
    subgraph Infrastructure
        AWS_S3[(AWS S3 Bucket)]:::aws
        AWS_EC2[AWS EC2]:::aws
        AWS_RDS[(AWS RDS)]:::aws
    end

    subgraph Data_Layer
        Training_Data[Training Data]:::data
        PostgreSQL_DB[(PostgreSQL DB)]:::database
        click PostgreSQL_DB "https://github.com/MohammmedAb/MLOps-Pipeline-Bike-Trip-Duration/blob/main/database/sql.py"
    end

    subgraph Training_and_Experimentation
        MLflow[MLflow Server]:::mlflow
        subgraph Prefect_Flows
            Training_Flow[Training Flow]:::prefect
            Batch_Metrics[Batch Metrics Calculation]:::prefect
            click Training_Flow "https://github.com/MohammmedAb/MLOps-Pipeline-Bike-Trip-Duration/blob/main/orchestration/train.py"
            click Batch_Metrics "https://github.com/MohammmedAb/MLOps-Pipeline-Bike-Trip-Duration/blob/main/orchestration/batch_metrics_calculation.py"
        end
        Notebooks[Data & Model Exploration]:::notebook
        click Notebooks "https://github.com/MohammmedAb/MLOps-Pipeline-Bike-Trip-Duration/blob/main/notebooks/data_and_model_exploration.ipynb"
    end

    subgraph Serving_Layer
        Flask_API[Flask API]:::api
        Docker[Docker Container]:::docker
        Gunicorn[Gunicorn Server]:::server
        API_Tests[API Tests]:::test
        click Flask_API "https://github.com/MohammmedAb/MLOps-Pipeline-Bike-Trip-Duration/blob/main/web/predict.py"
        click API_Tests "https://github.com/MohammmedAb/MLOps-Pipeline-Bike-Trip-Duration/blob/main/web/test.py"
        click Docker "https://github.com/MohammmedAb/MLOps-Pipeline-Bike-Trip-Duration/tree/main/Dockerfile"
    end

    subgraph Monitoring
        Evidently[Evidently Metrics]:::monitoring
        Grafana[Grafana Dashboards]:::monitoring
        MonitoringDB[(Monitoring DB)]:::database
        Monitoring_Tests[Monitoring Tests]:::test
        click Evidently "https://github.com/MohammmedAb/MLOps-Pipeline-Bike-Trip-Duration/blob/main/monitoring/dummy_metrics.py"
        click Grafana "https://github.com/MohammmedAb/MLOps-Pipeline-Bike-Trip-Duration/blob/main/config/grafana_dashboards.yaml"
        click Monitoring_Tests "https://github.com/MohammmedAb/MLOps-Pipeline-Bike-Trip-Duration/blob/main/notebooks/monitoring_test.ipynb"
    end

    %% Infrastructure Connections
    AWS_S3 --> Training_Data
    AWS_RDS --> PostgreSQL_DB
    AWS_EC2 --> MLflow
    AWS_EC2 --> Grafana
    AWS_EC2 --> MonitoringDB

    %% Data Flow
    Training_Data --> Training_Flow
    Training_Flow --> MLflow
    MLflow --> Flask_API
    Flask_API --> PostgreSQL_DB
    PostgreSQL_DB --> Batch_Metrics
    Batch_Metrics --> Evidently
    Evidently --> MonitoringDB
    MonitoringDB --> Grafana

    %% Service Orchestration
    Docker --> Flask_API
    Docker --> Gunicorn
    Flask_API --> API_Tests
    Evidently --> Monitoring_Tests

    %% Styling
    classDef aws fill:#232F3E,stroke:#FF9900,stroke-width:2px,color:white
    classDef data fill:#E8F5E9,stroke:#2E7D32,stroke-width:2px
    classDef database fill:#B2EBF2,stroke:#0097A7,stroke-width:2px
    classDef mlflow fill:#E1BEE7,stroke:#7B1FA2,stroke-width:2px
    classDef prefect fill:#FFCCBC,stroke:#E64A19,stroke-width:2px
    classDef api fill:#BBDEFB,stroke:#1976D2,stroke-width:2px
    classDef docker fill:#CFD8DC,stroke:#455A64,stroke-width:2px
    classDef server fill:#F5F5F5,stroke:#616161,stroke-width:2px
    classDef monitoring fill:#FFE0B2,stroke:#F57C00,stroke-width:2px
    classDef test fill:#F8BBD0,stroke:#C2185B,stroke-width:2px
    classDef notebook fill:#D1C4E9,stroke:#512DA8,stroke-width:2px
```
