# aditya-7-7/sales-forecasting-mlops-pipeline architecture

> Architecture diagram of the GitHub repository aditya-7-7/sales-forecasting-mlops-pipeline, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-09-21). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/aditya-7-7/sales-forecasting-mlops-pipeline
- Repository: https://github.com/aditya-7-7/sales-forecasting-mlops-pipeline

## Overview

##### Project Type & Purpose
- This is a full-stack MLOps platform for Sales Forecasting. It uses Apache Airflow (via Astronomer) to orchestrate an end-to-end ML pipeline, MLflow for experiment tracking, MinIO for artifact storage, ensemble models (XGBoost + LightGBM) for prediction, and a Streamlit-based UI for real-time inference.

##### File Structure Highlights & Architectural Clues
- **Top-Level**
- Dockerfile, docker-compose.override.yml → containerized services
- .astro/ & dags/ → Airflow DAG definitions
- / → reusable Python modules (data validation, feature engineering, ML model logic, utilities)
- ui/ → Streamlit inference app + helper scripts
- tests/ → unit tests for DAGs and core logic
- **Patterns**
- Modular code under include/ suggests clear separation of pipeline stages
- Container per service (Airflow Scheduler/Webserver, MLflow, MinIO, Streamlit)
- CI/testing focused on DAG integrity and core modules

##### Core Components
A. Orchestration Layer
- Astronomer/Airflow Service (Scheduler, Webserver, Workers)
- DAG: `sales_forecasting_training.py`
B. Pipeline Modules (Python Tasks invoked by Airflow)
1. Data Generation & Validation
- `include/utils/data_generator.py`
- `include/data_validation/validators.py`
2. Feature Engineering
- `include/feature_engineering/feature_pipeline.py`
3. Model Training & Ensemble
- `include/ml_models/train_models.py`
- `include/ml_models/ensemble_model.py`
- `include/ml_models/model_visualization.py`
4. Experiment Tracking & Artifact Storage
- MLflow Server (tracking, registry)
- MinIO S3 (artifact bucket)
C. Inference & Visualization
- Streamlit App (`ui/inference_app.py`)
- Model-loading utilities (`ui/utils/*.py`)
D. Infrastructure & DevOps
- Docker Compose network defining Airflow, MLflow, MinIO, Streamlit
- Configuration files (`include/config/ml_config.yaml`, .astro/config.yaml)

##### Interactions & Data Flow
1. **Airflow** triggers DAG on schedule or manually.
2. **Task A** uses Data Generator → writes synthetic data.
3. **Task B** runs Validators → ensures data quality.
4. **Task C** performs Feature Engineering → outputs feature store.
5. **Task D** trains XGBoost & LightGBM → logs metrics/artifacts to MLflow & uploads models to MinIO.
6. **Task E** runs ensemble logic → final model artifact.
7. **MLflow UI** available for browsing experiments.
8. **MinIO Console** for raw artifacts.
9. **Streamlit UI** calls MLflow/MinIO to load models, runs `include/feature_engineering` logic on input data, shows predictions & plots.

##### Architectural Patterns & Principles
- Pipeline-as-code: each stage is a standalone Python module invoked by Airflow
- Modular layering: separation of concerns (data, features, training, inference)
- Service isolation via containers (Airflow, MLflow, MinIO, UI)
- Experiment reproducibility: configuration driven, MLflow versioning, S3 storage

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Orchestration Layer
    subgraph "Orchestration Layer"
        direction TB
        AF[/"Airflow Scheduler\nWebserver\nWorkers"/]:::orchestration
        DAG["sales_forecasting_training.py\n(DAG Definition)"]:::orchestration
    end

    click DAG "https://github.com/aditya-7-7/sales-forecasting-mlops-pipeline/blob/main/dags/sales_forecasting_training.py"
    click AF "https://github.com/aditya-7-7/sales-forecasting-mlops-pipeline/blob/main/.astro/config.yaml"

    %% Pipeline Modules
    subgraph "Pipeline Tasks"
        direction TB
        DG["Data Generator\ndata_generator.py"]:::data
        DV["Data Validation\nvalidators.py"]:::data
        FE["Feature Engineering\nfeature_pipeline.py"]:::data
        MT["Model Training\ntrain_models.py"]:::modeling
        EM["Ensemble Logic\nensemble_model.py"]:::modeling
    end

    click DG "https://github.com/aditya-7-7/sales-forecasting-mlops-pipeline/blob/main/include/utils/data_generator.py"
    click DV "https://github.com/aditya-7-7/sales-forecasting-mlops-pipeline/blob/main/include/data_validation/validators.py"
    click FE "https://github.com/aditya-7-7/sales-forecasting-mlops-pipeline/blob/main/include/feature_engineering/feature_pipeline.py"
    click MT "https://github.com/aditya-7-7/sales-forecasting-mlops-pipeline/blob/main/include/ml_models/train_models.py"
    click EM "https://github.com/aditya-7-7/sales-forecasting-mlops-pipeline/blob/main/include/ml_models/ensemble_model.py"

    %% Storage & Tracking
    subgraph "Storage & Tracking"
        direction TB
        MLFS["MLflow Server\ntracking & registry"]:::storage
        MLDB[(MLflow Backend DB)]:::storage
        MIO[(MinIO S3 Storage\nartifact bucket)]:::storage
    end

    click MLFS "https://github.com/aditya-7-7/sales-forecasting-mlops-pipeline/blob/main/Dockerfile.mlflow"
    click MLDB "https://github.com/aditya-7-7/sales-forecasting-mlops-pipeline/blob/main/include/utils/mlflow_utils.py"
    click MLDB "https://github.com/aditya-7-7/sales-forecasting-mlops-pipeline/blob/main/include/utils/mlflow_s3_utils.py"
    click MLDB "https://github.com/aditya-7-7/sales-forecasting-mlops-pipeline/blob/main/include/utils/s3_verification.py"
    click MIO "https://github.com/aditya-7-7/sales-forecasting-mlops-pipeline/blob/main/include/config/ml_config.yaml"

    %% Inference & UI
    subgraph "Inference & Visualization"
        direction TB
        STREAM["Streamlit Service\nInference App"]:::ui
        UIAPP["inference_app.py"]:::ui
        EML["Ensemble Loader\nensemble_model_standalone.py"]:::ui
        SML["Model Loader\nsimple_model_loader.py"]:::ui
        PRED["Prediction Logic\nsimple_predictor.py"]:::ui
    end

    click UIAPP "https://github.com/aditya-7-7/sales-forecasting-mlops-pipeline/blob/main/ui/inference_app.py"
    click EML "https://github.com/aditya-7-7/sales-forecasting-mlops-pipeline/blob/main/ui/utils/ensemble_model_standalone.py"
    click SML "https://github.com/aditya-7-7/sales-forecasting-mlops-pipeline/blob/main/ui/utils/simple_model_loader.py"
    click PRED "https://github.com/aditya-7-7/sales-forecasting-mlops-pipeline/blob/main/ui/utils/simple_predictor.py"
    click STREAM "https://github.com/aditya-7-7/sales-forecasting-mlops-pipeline/tree/main/ui/Dockerfile"
    click STREAM "https://github.com/aditya-7-7/sales-forecasting-mlops-pipeline/blob/main/ui/Dockerfile.streamlit"
    click STREAM "https://github.com/aditya-7-7/sales-forecasting-mlops-pipeline/blob/main/ui/entrypoint.sh"

    %% CI & DevOps
    subgraph "Infrastructure & DevOps"
        direction TB
        DOCK[/"docker-compose.override.yml"/]:::infra
        BASE[/"Dockerfile"/]:::infra
        IGNORE[".airflowignore"]:::infra
        CI1["dag_integrity_exceptions.txt"]:::infra
        CI2["test_dag_integrity_default.py"]:::infra
        REQ["requirements.txt\nui/requirements.txt\npackages.txt"]:::infra
    end

    click DOCK "https://github.com/aditya-7-7/sales-forecasting-mlops-pipeline/blob/main/docker-compose.override.yml"
    click BASE "https://github.com/aditya-7-7/sales-forecasting-mlops-pipeline/tree/main/Dockerfile"
    click IGNORE "https://github.com/aditya-7-7/sales-forecasting-mlops-pipeline/blob/main/dags/.airflowignore"
    click CI1 "https://github.com/aditya-7-7/sales-forecasting-mlops-pipeline/blob/main/.astro/dag_integrity_exceptions.txt"
    click CI2 "https://github.com/aditya-7-7/sales-forecasting-mlops-pipeline/blob/main/.astro/test_dag_integrity_default.py"
    click REQ "https://github.com/aditya-7-7/sales-forecasting-mlops-pipeline/blob/main/requirements.txt"
    click REQ "https://github.com/aditya-7-7/sales-forecasting-mlops-pipeline/blob/main/ui/requirements.txt"
    click REQ "https://github.com/aditya-7-7/sales-forecasting-mlops-pipeline/blob/main/packages.txt"

    %% External UIs
    User["User"]:::external
    AFUI["Airflow UI"]:::external
    MLUI["MLflow UI"]:::external
    MIOUI["MinIO Console"]:::external
    STUI["Streamlit UI"]:::external

    %% Connections
    User --> AFUI
    User --> MLUI
    User --> MIOUI
    User --> STUI

    AF --> DAG
    DAG --> DG
    DAG --> DV
    DAG --> FE
    DAG --> MT
    DAG --> EM

    DG --> DV --> FE --> MT --> EM

    MT -->|metrics/logs| MLFS
    MT -->|artifacts/models| MIO
    EM -->|final_model.pkl| MIO

    STREAM -->|load models| MLFS
    STREAM -->|load artifacts| MIO
    STREAM --> EML
    STREAM --> SML
    STREAM --> PRED

    %% Styles
    classDef orchestration fill:#E0F7FA,stroke:#0288D1,color:#01579B
    classDef data fill:#C8E6C9,stroke:#388E3C,color:#1B5E20
    classDef modeling fill:#FFCC80,stroke:#F57C00,color:#E65100
    classDef storage fill:#E1BEE7,stroke:#8E24AA,color:#4A148C
    classDef ui fill:#B2EBF2,stroke:#00796B,color:#004D40
    classDef infra fill:#F5F5F5,stroke:#9E9E9E,color:#424242
    classDef external fill:#FFFFFF,stroke:#000000,color:#000000,stroke-dasharray: 5 5
```
