# gokumohandas/mlops-course architecture

> Architecture diagram of the GitHub repository gokumohandas/mlops-course, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-16). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/gokumohandas/mlops-course
- Repository: https://github.com/gokumohandas/mlops-course

## Overview

##### Main Components
- Data Management: This includes the datasets (CSV files in the “datasets” directory) and any utilities for data loading (see madewithml/data.py) plus tests in tests/data.
- Core ML Workloads: The “madewithml” folder contains key scripts for training (train.py), tuning (tune.py), evaluation (evaluate.py), inference/prediction (predict.py) and serving (serve.py) as well as supporting modules (config.py, models.py, utils.py). Also note that “notebooks” provide an interactive environment for exploring these functions.
- Experiment Tracking: The README covers the integration with MLflow for tracking experiments and model registry setup.
- Deployment Infrastructure: The “deploy” directory holds configuration files and scripts (cluster compute/environment descriptions, job definitions, service definitions) for deploying workloads and model serving.
- CI/CD & Automation: The “.github/workflows” files, pre-commit hooks, Makefile, and other configuration files enable automated testing, job submission, and service rollout via GitHub Actions.

##### Relationships and Interactions
- A clear flow exists from data ingestion (using CSVs) → model training (train.py) → hyperparameter tuning (tune.py) → evaluation (evaluate.py) and inference (predict.py) → model serving (serve.py).
- The MLflow tracking system is used to record experiments that are produced across training, tuning, and evaluation workflows.
- Deployment configurations (in deploy/) use Anyscale and Ray to run jobs and serve models in production.
- CI/CD workflows (in “.github/workflows”) trigger jobs (using workloads/job configurations) and serve deployments when merging into the main branch.

##### Important Architectural Patterns and Design Principles
- Modularity: Each major ML function is implemented as a separate Python script/module. This modularity allows for independent testing (see tests/code, tests/data, and tests/model) and easy extension.
- Separation of Concerns in CI/CD: There’s a clear separation between the development (local notebooks and scripts) and production configurations (deployment YAML files and CI/CD workflows).
- Scalability via Cluster Configuration: The use of cluster environments, compute configs (deploy/cluster_compute.yaml, deploy/cluster_env.yaml) and orchestration of Anyscale jobs/services indicates emphasis on scaling ML workloads.
- Experiment Reproducibility: Usage of MLflow and containerized configurations ensures reproducibility of results across environments.

##### Relevant Technologies, Frameworks, and Tools
- Python (with modular scripts for each ML stage).
- Jupyter Notebooks for experimentation.
- MLflow for experiment tracking.
- Ray for distributed model serving and workload management.
- Anyscale for cluster management and compute orchestration.
- GitHub Actions, pre-commit tooling, and Docker-like deployment configs (YAML files in deploy) for CI/CD integration.
- Pytest is used for testing various components.

##### Putting It All Together
- A section for “Deployment & Orchestration” that details both CI/CD (GitHub Actions workflows, pre-commit) and cluster management components (deploy/ directory, Anyscale & Ray orchestration).

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% ML Workflow Lane
    subgraph "ML Workflow"
        %% Data Management Subgraph
        subgraph "Data Management"
            DM1("Datasets"):::mlWorkflow
            DM2("Data Loading (madewithml/data.py)"):::mlWorkflow
            DM3("Data Tests"):::mlWorkflow
        end

        T("Training (madewithml/train.py)"):::mlWorkflow
        TT("Tuning (madewithml/tune.py)"):::mlWorkflow
        E("Evaluation (madewithml/evaluate.py)"):::mlWorkflow
        I("Inference/Prediction (madewithml/predict.py)"):::mlWorkflow
        S("Serving (madewithml/serve.py)"):::mlWorkflow

        %% Supporting Modules Subgraph
        subgraph "Supporting Modules"
            SM1("Config (madewithml/config.py)"):::mlWorkflow
            SM2("Models (madewithml/models.py)"):::mlWorkflow
            SM3("Utils (madewithml/utils.py)"):::mlWorkflow
        end

        N("Notebooks (notebooks/)"):::mlWorkflow
        MT("MLflow (ExperimentTracking)"):::external
    end

    %% Deployment & CI/CD Lane
    subgraph "Deployment & CI/CD"
        subgraph "CI/CD Pipeline"
            GH("GitHub Actions (.github/workflows/)"):::deployment
            PC("Pre-commit Hooks (.pre-commit-config.yaml)"):::deployment
            MF("Makefile"):::deployment
        end
        subgraph "Deployment Infrastructure"
            DC("Deployment Configs (deploy/)"):::deployment
            JD("Job Definitions (deploy/jobs/)"):::deployment
            SD("Service Definitions (deploy/services/)"):::deployment
        end
        CO("Cluster Orchestrator (Ray&Anyscale)"):::deployment
    end

    %% ML Workflow Relationships
    DM1 --> T
    DM2 --> T
    T --> TT
    TT --> E
    E --> I
    I --> S

    %% Logging to MLflow
    T -->|"logs"| MT
    E -->|"logs"| MT

    %% Supporting Modules influence
    SM1 --> T
    SM2 --> TT
    SM3 --> E

    %% Notebooks interaction
    N --> T
    N --> E

    %% Deployment Relationships
    GH -->|"triggers"| DC
    PC -->|"triggers"| DC
    MF -->|"triggers"| DC
    S -->|"deployed"| DC
    DC --> JD
    DC --> SD
    DC --> CO

    %% Styles
    classDef mlWorkflow fill:#E0F7FA,stroke:#000,stroke-width:2px;
    classDef deployment fill:#F9E79F,stroke:#000,stroke-width:2px;
    classDef external fill:#D2B4DE,stroke:#000,stroke-width:2px;

    %% Click Events for Data Management
    click DM1 "https://github.com/gokumohandas/mlops-course/tree/main/datasets/"
    click DM2 "https://github.com/gokumohandas/mlops-course/blob/main/madewithml/data.py"
    click DM3 "https://github.com/gokumohandas/mlops-course/tree/main/tests/data/"

    %% Click Events for Core ML Workloads
    click T "https://github.com/gokumohandas/mlops-course/blob/main/madewithml/train.py"
    click TT "https://github.com/gokumohandas/mlops-course/blob/main/madewithml/tune.py"
    click E "https://github.com/gokumohandas/mlops-course/blob/main/madewithml/evaluate.py"
    click I "https://github.com/gokumohandas/mlops-course/blob/main/madewithml/predict.py"
    click S "https://github.com/gokumohandas/mlops-course/blob/main/madewithml/serve.py"

    %% Click Events for Supporting Modules
    click SM1 "https://github.com/gokumohandas/mlops-course/blob/main/madewithml/config.py"
    click SM2 "https://github.com/gokumohandas/mlops-course/blob/main/madewithml/models.py"
    click SM3 "https://github.com/gokumohandas/mlops-course/blob/main/madewithml/utils.py"

    %% Click Event for Notebooks
    click N "https://github.com/gokumohandas/mlops-course/tree/main/notebooks/"

    %% Click Events for Deployment Infrastructure
    click GH "https://github.com/gokumohandas/mlops-course/tree/main/.github/workflows/"
    click PC "https://github.com/gokumohandas/mlops-course/blob/main/.pre-commit-config.yaml"
    click MF "https://github.com/gokumohandas/mlops-course/tree/main/Makefile"
    click DC "https://github.com/gokumohandas/mlops-course/tree/main/deploy/"
    click JD "https://github.com/gokumohandas/mlops-course/tree/main/deploy/jobs/"
    click SD "https://github.com/gokumohandas/mlops-course/tree/main/deploy/services/"
```
