# hurryingauto3/ece-gy-9183-mlops-project architecture

> Architecture diagram of the GitHub repository hurryingauto3/ece-gy-9183-mlops-project, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-13). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/hurryingauto3/ece-gy-9183-mlops-project
- Repository: https://github.com/hurryingauto3/ece-gy-9183-mlops-project

## Overview

##### Project type and purpose
- This is a cloud-native, full-stack MLOps system (“AgriYield”) for end-to-end crop yield prediction.
- It ingests large satellite/weather datasets (CropNet), runs ETL, trains deep learning models (LSTM+TCN) on GPU nodes, tracks experiments in MLflow, serves features and predictions via FastAPI microservices, and monitors system and model health with Prometheus/Grafana and Locust. Infrastructure is managed via Terraform, Ansible, K3s (lightweight Kubernetes), and GitOps (ArgoCD).

##### File structure
- Top-level directories:
- chameleon_devops—Terraform, Ansible, K8s manifests, Helm charts, ArgoCD apps (infrastructure as code + GitOps)
- etl_pipeline—Dockerized extract & transform jobs
- feature_serving—FastAPI service to retrieve pre-computed features
- model_training—PyTorch training scripts, Jupyter notebooks, Ray support
- model_serving—FastAPI microservice for inference with MLflow fallback
- dashboard & tutorials—User-facing Streamlit/Gradio dashboards and example notebooks
- monitoring—Prometheus/Grafana configs, Locust load-test scripts
- mlflow—Compose file for MLflow server and MinIO object store
- Key inf rastructure files: Terraform .tf under chameleon_devops/terraform; Ansible playbooks & roles under chameleon_devops/ansible; K8s manifests under chameleon_devops/k8s; Docker-compose files at root and in subdirs.

##### Pull insights from the README
- Core components: Data Lake (persistent storage + MinIO), ETL pipeline, Feature Service, Training Platform (GPU node, Ray, PyTorch, MLflow), Model Serving API, Dashboards, Monitoring stack, CI/CD/GitOps.
- Infrastructure: Two VMs on Chameleon Cloud (services_node, gpu_node) joined in a K3s cluster. Floating IPs, block storage, GPU reservation.
- Deployment & automation: Terraform + Ansible for infra; ArgoCD for continuous deployment; Docker images built via CI; canary/staging/production overlays.

##### Main components & interactions
- Cloud/IaaS layer: Two VMs (services_node, gpu_node) with Attached Block Storage (CropNet data, MLflow artifacts)
- Kubernetes (K3s) cluster: Master + worker nodes on those VMs
- Data services:
- MinIO (S3-compatible) for object storage
- Data Lake (persistent volumes hosting raw CropNet files)
- ETL Pipeline: Dockerized Extract and Transform jobs (etl_pipeline/extract and etl_pipeline/transform)
- Feature Service: FastAPI (feature_serving)
- Model Training:
- PyTorch LSTM+TCN containers (model_training)
- Ray cluster for distributed jobs (optional)
- MLflow server & UI (mlflow/docker-compose)
- Model Registry: MLflow Model Registry
- Model Serving: FastAPI microservice (model_serving) with MLflow loader & dummy fallback
- Dashboard/UI: Streamlit/Gradio (dashboard/app.py)
- CI/CD/GitOps: GitHub → ArgoCD → K8s deployments (chameleon_devops/k8s, charts, manifests)
- Monitoring & Testing:
- Prometheus (scraping K3s, services)
- Grafana dashboards (monitoring/grafana, provisioning files)
- Locust load tests (monitoring/locustfile.py)
1. Offline: CropNet files → ETL extract container → transformed features → Data Lake / MinIO
2. Feature Service queries Data Lake / MinIO → returns feature vectors
3. Model Training containers pull features via Feature Service or directly from Data Lake → run training on GPU node → log metrics/artifacts to MLflow/MinIO
4. Upon promotion (via promote_model.py or CI), trained model version moves to “Production” in MLflow registry
5. Model Serving API pulls production model from MLflow registry on startup/inference → serves /predict endpoints
6. Dashboard UI calls Feature Service and Model Serving API to display predictions vs. truth
7. Monitoring: Prometheus scrapes metrics from ETL pods, Feature Service, Model Serving, Ray, MLflow → Grafana visualizes; Locust periodically sends traffic to /predict and /features
8. CI/CD: GitHub updates → ArgoCD syncs to K8s cluster → Kubernetes applies new Docker images / manifests in staging → canary → production
c. Architectural patterns & design principles:
- Microservices (ETL, Feature, Training, Serving, Dashboard each in its own container)
- Containerization + Kubernetes for orchestration (K3s)
- GitOps (ArgoCD) + IaC (Terraform, Ansible)
- Model registry and versioning (MLflow)
- Infrastructure layering: IaaS → Kubernetes → Services
- Resiliency with dummy fallback for model serving; separate staging/canary/production overlays
d. Technologies & framework callouts:
- Terraform, Ansible, K3s, ArgoCD, Helm
- Docker / Docker-Compose
- Python, FastAPI, PyTorch, Ray
- MLflow, MinIO
- Prometheus, Grafana, Locust
- Streamlit/Gradio for dashboard

##### Detail level
- Major microservices and their interactions
- Infrastructure provisioning paths (Terraform/Ansible → VMs → K3s cluster)
- MLflow registry transitions (None → Staging → Production)
- Canary/staging/production overlays in GitOps loop
- Don’t clutter with every file—aggregate by functional directory

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Infrastructure Layer
    subgraph "Cloud / IaaS" 
        direction TB
        Cloud["Chameleon Cloud"]:::infra
        VM1["services_node VM"]:::infra
        VM2["gpu_node VM"]:::infra
        Cloud --> VM1
        Cloud --> VM2
        subgraph "K3s Cluster" 
            direction TB
            VM1 --> K3sMaster["K3s Master"]:::infra
            VM1 --> K3sWorker["K3s Worker"]:::infra
            VM2 --> K3sWorkerGPU["K3s Worker (GPU)"]:::infra
        end
    end

    %% Data Layer
    subgraph "Data Layer" 
        direction TB
        DataLake["Data Lake\n(persistent storage)"]:::data
        MinIO["MinIO Object Store"]:::data
    end

    %% ETL Pipeline
    subgraph "ETL Pipeline" 
        direction TB
        Extract["Extract Job"]:::service
        Transform["Transform Job"]:::service
    end

    %% Feature Service
    FeatureService["Feature Service\n(FastAPI)"]:::service

    %% Model Training
    subgraph "Model Training" 
        direction TB
        Training["Training Container\n(PyTorch + Ray)"]:::service
        Promote["Promotion Script"]:::service
    end

    %% Model Registry
    subgraph "MLflow" 
        direction TB
        MLflowServer["MLflow Server & UI"]:::service
        ModelRegistry["Model Registry"]:::data
    end

    %% Model Serving
    subgraph "Model Serving" 
        direction TB
        ModelAPI["Model Serving API\n(FastAPI)"]:::service
        Loader["MLflow Loader Module"]:::service
        Predict["Prediction Logic"]:::service
    end

    %% Dashboard
    subgraph "Dashboard / UI" 
        direction TB
        DashboardApp["Dashboard App\n(Streamlit/Gradio)"]:::service
        DashboardAPI["Dashboard API Service"]:::service
    end

    %% CI/CD / GitOps
    subgraph "CI/CD & GitOps" 
        direction TB
        GitHub["GitHub Repo"]:::cicd
        ArgoCD["ArgoCD"]:::cicd
        Terraform["Terraform IaC"]:::cicd
        Ansible["Ansible Playbooks"]:::cicd
    end

    %% Monitoring & Testing
    subgraph "Monitoring & Testing" 
        direction TB
        Prometheus["Prometheus"]:::monitor
        Grafana["Grafana"]:::monitor
        Locust["Locust Load Testing"]:::monitor
    end

    %% Data Flows
    DataLake --> Extract
    Extract --> Transform
    Transform --> DataLake
    DataLake --> FeatureService
    FeatureService --> Training
    DataLake --> Training
    Training --> MLflowServer
    MLflowServer --> ModelRegistry
    Promote --> ModelRegistry
    ModelRegistry --> Loader
    Loader --> ModelAPI
    ModelAPI --> Predict
    FeatureService --> DashboardApp
    ModelAPI --> DashboardApp

    %% Monitoring Flows
    Prometheus --> Grafana
    ModelAPI --> Prometheus
    FeatureService --> Prometheus
    Extract --> Prometheus
    Training --> Prometheus
    MLflowServer --> Prometheus
    Locust --> FeatureService
    Locust --> ModelAPI

    %% CI/CD Flows
    GitHub --> ArgoCD
    ArgoCD --> K3sMaster
    Terraform --> VM1
    Terraform --> VM2
    Ansible --> VM1
    Ansible --> VM2

    %% Click Events
    click DataLake "https://github.com/hurryingauto3/ece-gy-9183-mlops-project/tree/main/data/data_lake"
    click Extract "https://github.com/hurryingauto3/ece-gy-9183-mlops-project/blob/main/etl_pipeline/extract/extract.py"
    click Transform "https://github.com/hurryingauto3/ece-gy-9183-mlops-project/blob/main/etl_pipeline/transform/transform.py"
    click FeatureService "https://github.com/hurryingauto3/ece-gy-9183-mlops-project/blob/main/feature_serving/main.py"
    click Training "https://github.com/hurryingauto3/ece-gy-9183-mlops-project/blob/main/model_training/train.py"
    click Promote "https://github.com/hurryingauto3/ece-gy-9183-mlops-project/blob/main/model_training/promote_model.py"
    click MLflowServer "https://github.com/hurryingauto3/ece-gy-9183-mlops-project/blob/main/mlflow/docker-compose-mlflow.yaml"
    click MLflowServer "https://github.com/hurryingauto3/ece-gy-9183-mlops-project/blob/main/mlflow/minio_config.json"
    click ModelAPI "https://github.com/hurryingauto3/ece-gy-9183-mlops-project/blob/main/model_serving/app.py"
    click Loader "https://github.com/hurryingauto3/ece-gy-9183-mlops-project/blob/main/model_serving/mlflow_loader.py"
    click Predict "https://github.com/hurryingauto3/ece-gy-9183-mlops-project/blob/main/model_serving/predict.py"
    click DashboardApp "https://github.com/hurryingauto3/ece-gy-9183-mlops-project/blob/main/dashboard/app.py"
    click DashboardAPI "https://github.com/hurryingauto3/ece-gy-9183-mlops-project/blob/main/dashboard/api_service.py"
    click Prometheus "https://github.com/hurryingauto3/ece-gy-9183-mlops-project/blob/main/monitoring/prometheus/prometheus.yml"
    click Grafana "https://github.com/hurryingauto3/ece-gy-9183-mlops-project/tree/main/monitoring/grafana"
    click Locust "https://github.com/hurryingauto3/ece-gy-9183-mlops-project/blob/main/monitoring/locustfile.py"
    click Terraform "https://github.com/hurryingauto3/ece-gy-9183-mlops-project/blob/main/chameleon_devops/terraform/main.tf"
    click Ansible "https://github.com/hurryingauto3/ece-gy-9183-mlops-project/tree/main/chameleon_devops/ansible/playbooks"
    click ArgoCD "https://github.com/hurryingauto3/ece-gy-9183-mlops-project/tree/main/chameleon_devops/k8s"
    click ArgoCD "https://github.com/hurryingauto3/ece-gy-9183-mlops-project/tree/main/chameleon_devops/charts"

    %% Styles
    classDef infra fill:#dddddd,stroke:#888888,color:#333333
    classDef data fill:#a3c4f3,stroke:#03396c,color:#03396c
    classDef service fill:#98d897,stroke:#006400,color:#005000
    classDef cicd fill:#f7a072,stroke:#cc5500,color:#663300
    classDef monitor fill:#c49cde,stroke:#6a0dad,color:#4b0082
```
