# com6012/scalableml architecture

> Architecture diagram of the GitHub repository com6012/scalableml, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-01). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/com6012/scalableml
- Repository: https://github.com/com6012/scalableml

## Overview

- Code – contains Python scripts (e.g., LogMining100.py, LogMiningBig.py) that serve as practical examples or lab exercises.
- Data – provides various sample datasets (e.g., Advertising.csv, MovieLens100k.data, NASA logs, iris.csv, spambase.data, etc.) used in practical sessions.
- HPC – includes scripts for running batch jobs (e.g., sample_batch.sh) on HPC clusters.
- Lab notes – Markdown files (e.g., Lab 1 – Introduction to Spark and HPC.md up through Lab 8, etc.) that describe and guide through different aspects of scalable machine learning using Spark.
- Output – holds sample results from running labs.
- Slides – provides lecture slides and overviews of the modules.
- README – contextualizes the project as a part of a course curriculum (Spring 2025) with introductory material, session breakdown, and acknowledgements.

##### Main Components
a. Educational Modules Layer:
– Lab session files (the Markdown files with instructions for each session)
– Lecture slides
b. Code Component:
– Python scripts in the Code directory that implement practical examples (e.g., log mining operations with Spark).
c. Data Repository:
– Various datasets used for exercises (advertising, movie ratings, NASA logs, etc.).
d. Execution Platform:
– HPC cluster (illustrated through the HPC directory and sample batch scripts) where students access resources via SSH/VPN and run Spark jobs.
e. Processing Layer:
– Apache Spark (PySpark 3.5.4 as specified) that underpins the scalable machine learning operations.
f. Output/Results:
– Output directory that logs results and outputs from the labs.

##### Relationships and Interactions
– The “User” (student or instructor) connects to the HPC cluster (using SSH with MFA) to submit jobs.
– Upon submission, the HPC system schedules Spark jobs defined in the Code scripts.
– Apache Spark, running on the cluster, processes the datasets (from the Data folder) using distributed computations.
– The processed results are stored in the Output folder.
– Lab notes and slides provide instructions and context throughout the process.

##### Architectural Patterns and Design Principles
– Modular design: Separation between teaching material (labs and slides), real code (Python scripts), data, and the execution environment (HPC + Spark).
– Client–Server model: Students (clients) interact via SSH/VPN with the remote HPC cluster (server) to run distributed Spark jobs.
– Educational orchestration: The flow from instructions (lab notes) to execution (Spark jobs on HPC) to result analysis forms a tightly integrated learning pipeline.

##### Technologies and Tools to Highlight
– Apache Spark (specifically PySpark 3.5.4) for distributed processing.
– HPC Cluster environment and accompanying shell scripts for job management.
– Python for the code developed in labs.
– Standard datasets that feed into machine learning exercises.

– Clear labels for each component (e.g., “Lab Sessions”, “Python Code”, “Datasets”, “HPC Cluster”, “Apache Spark Processing”, “Output Results”).
- From “User” to “HPC” (for job submission),
- From “HPC” to “Spark” (job processing across the cluster),
– Annotated connection lines where HPC scripts in the HPC directory trigger the Spark jobs, and where the lab instructions tie the components together in a learning pipeline.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Nodes outside subgraphs
    User["User (Student/Instructor)"]:::user

    %% Educational Modules Subgraph
    subgraph "Educational Modules"
        subgraph "Lab Sessions"
            Lab1["Lab 1 - Introduction to Spark and HPC"]:::edu
            Lab2["Lab 2 - RDD, DataFrame, ML pipeline, and parallelization"]:::edu
            Lab3["Lab 3 - Spark configuration and scalable logistic regression"]:::edu
            Lab4["Lab 4 - Scalable Generalized Linear Models"]:::edu
            Lab5["Lab 5- Scalable Decision trees"]:::edu
            Lab6["Lab 6 - Scalable neural networks"]:::edu
            Lab7["Lab 7 - Scalable k-means clustering"]:::edu
            Lab8["Lab 8 - Sclable matrix factorization and PCA"]:::edu
        end
        LectureSlides["Lecture Slides"]:::edu
    end

    %% Development Components Subgraph
    subgraph "Development Components"
        PythonCode["Python Scripts"]:::code
        Datasets["Datasets"]:::data
    end

    %% Execution Environment Subgraph
    subgraph "Execution Environment"
        HPC["HPC Cluster"]:::exec
        Spark["Apache Spark Processing (PySpark 3.5.4)"]:::spark
        Output["Output Results"]:::exec
    end

    %% Connections / Data Flow
    User -->|"SSH/MFA"| HPC
    HPC -->|"Triggers"| Spark
    Datasets -->|"InputData"| Spark
    PythonCode -->|"SparkJobCode"| Spark
    Spark -->|"StoresOutputs"| Output
    LectureSlides -->|"Guides"| HPC
    LectureSlides -->|"Guides"| PythonCode
    Lab1 -->|"Exercises"| PythonCode
    Lab1 -->|"Exercises"| HPC

    %% Styles for vertical orientation spacing
    linkStyle default stroke:#333,stroke-width:2px

    %% Click Events
    click Lab1 "https://github.com/com6012/scalableml/blob/master/Lab 1 - Introduction to Spark and HPC.md"
    click Lab2 "https://github.com/com6012/scalableml/blob/master/Lab 2 - RDD, DataFrame, ML pipeline, and parallelization.md"
    click Lab3 "https://github.com/com6012/scalableml/blob/master/Lab 3 - Spark configuration and scalable logistic regression.md"
    click Lab4 "https://github.com/com6012/scalableml/blob/master/Lab 4 - Scalable Generalized Linear Models.md"
    click Lab5 "https://github.com/com6012/scalableml/blob/master/Lab 5- Scalable Decision trees.md"
    click Lab6 "https://github.com/com6012/scalableml/blob/master/Lab 6 - Scalable neural networks.md"
    click Lab7 "https://github.com/com6012/scalableml/blob/master/Lab 7 - Scalable k-means clustering.md"
    click Lab8 "https://github.com/com6012/scalableml/blob/master/Lab 8 - Sclable matrix factorization and PCA.md"
    click LectureSlides "https://github.com/com6012/scalableml/tree/master/Slides"
    click PythonCode "https://github.com/com6012/scalableml/tree/master/Code"
    click HPC "https://github.com/com6012/scalableml/blob/master/HPC/sample_batch.sh"
    click Datasets "https://github.com/com6012/scalableml/tree/master/Data"
    click Output "https://github.com/com6012/scalableml/tree/master/Output"

    %% Class Definitions for Styling
    classDef edu fill:#FFCCFF,stroke:#333,stroke-width:2px;
    classDef code fill:#CCFFFF,stroke:#333,stroke-width:2px;
    classDef data fill:#FFFFCC,stroke:#333,stroke-width:2px;
    classDef exec fill:#CCFFCC,stroke:#333,stroke-width:2px;
    classDef spark fill:#FFCC99,stroke:#333,stroke-width:2px;
    classDef user fill:#CCCCFF,stroke:#333,stroke-width:2px;
```
