# nicola-cecere/massively-parallel-ml architecture

> Architecture diagram of the GitHub repository nicola-cecere/massively-parallel-ml, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-11-01). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/nicola-cecere/massively-parallel-ml
- Repository: https://github.com/nicola-cecere/massively-parallel-ml

## Overview

##### Project Type & Purpose
- This is a Python+Spark research/demo codebase illustrating “massively parallel” implementations of two core ML algorithms:
– K-Means clustering on MNIST
– Logistic Regression for botnet detection
- It provides both centralized (single‐machine) and parallelized (Spark) variants, plus cross‐validation for the botnet project.

##### Key Components
a. Data Sources
– MNIST image files (raw pixel data)
– Botnet network‐flow dataset
b. Centralized Pipelines (control flow)
– Data reader/loader modules (read.py)
– Preprocessing modules (normalize.py, assignCluster.py)
– Training module (train.py / kmeans.py)
– Prediction / clustering modules (predict.py)
– Evaluation modules (accuracy.py)
– Jupyter notebooks tying it all together
c. Parallel Pipelines (Spark)
– Spark driver process (main.py)
– RDD/DataFrame creation from raw data
– Parallelized map/reduce or iterative steps for K-Means and Logistic Regression
– Spark executors that run bulk computations
– Aggregation and collection of results
d. Cross‐Validation (Botnet only)
– Data split and fold generation (get_block_data.py, transform.py)
– Training/evaluation per fold
– Aggregation of fold‐metrics
e. Supporting Artifacts
– setup.py / pyproject.toml for environment & packaging
– PDF reports & slides for documentation

##### Interactions & Data Flow
- Raw Data → Preprocessor → Training → Model Parameters → Predictor → Evaluator → Metric outputs
- In parallel mode, the “Training” and “Prediction” stages run distributed across Spark executors; the driver orchestrates.
- In cross‐validation, multiple parallel “fold” pipelines run sequentially or in parallel depending on your Spark setup.

##### Architectural Patterns & Principles
- Pipeline pattern: clear stages (ingest → preprocess → train → predict → eval)
- Separation of concerns: centralized vs. parallel code under distinct directories
- Master–worker model in Spark: one driver, many executors
- Modularity: each algorithm (K-Means, Logistic Regression) has its own set of modules

## Diagram (Mermaid)

```mermaid
graph TD
    %% Data Sources
    MNIST_DS[(MNIST Images)]:::datasource
    Botnet_DS[(Botnet CSVs)]:::datasource

    %% Centralized Pipeline
    subgraph "Centralized Pipeline"
        direction TB
        R1["MNIST Reader"]:::pythonModule
        P1["Cluster Assigner"]:::pythonModule
        T1["K-Means Trainer"]:::pythonModule
        Pr1["MNIST Predictor"]:::pythonModule
        E1["Accuracy Evaluator"]:::pythonModule
        N1["MNIST Centralized Notebook"]:::notebook

        R2["Botnet Preprocessor"]:::pythonModule
        N2["Botnet Normalizer"]:::pythonModule
        T2["LogReg Trainer"]:::pythonModule
        Pr2["Botnet Predictor"]:::pythonModule
        E2["Accuracy Evaluator"]:::pythonModule
        N3["Botnet Centralized Notebook"]:::notebook
    end

    %% Centralized MNIST Flow
    MNIST_DS --> R1 --> P1 --> T1 --> Pr1 --> E1 --> N1

    %% Centralized Botnet Flow
    Botnet_DS --> R2 --> N2 --> T2 --> Pr2 --> E2 --> N3

    %% Parallel (Spark) Pipeline
    subgraph "Parallel (Spark) Pipeline"
        direction TB
        SD1["MNIST Spark Driver"]:::spark
        Exec1["Spark Executors"]:::spark
        Col1["Results Collector"]:::spark
        N4["MNIST Parallel Notebook"]:::notebook

        SD2["Botnet Spark Driver"]:::spark
        Exec2["Spark Executors"]:::spark
        Col2["Results Collector"]:::spark
        N5["Botnet Parallel Notebook"]:::notebook
    end

    %% Parallel MNIST Flow
    MNIST_DS --> SD1 --> Exec1 --> Col1 --> N4

    %% Parallel Botnet Flow
    Botnet_DS --> SD2 --> Exec2 --> Col2 --> N5

    %% Cross-Validation Pipeline (Botnet)
    subgraph "Cross-Validation Pipeline (Botnet)"
        direction TB
        S3["Data Splitter"]:::pythonModule
        Tr3["Transformer"]:::pythonModule
        P3["Preprocessor"]:::pythonModule
        N6["Normalizer"]:::pythonModule
        T3["Trainer"]:::pythonModule
        Pr3["Predictor"]:::pythonModule
        E3["Evaluator"]:::pythonModule
        N7["Cross-Validation Notebook"]:::notebook
    end

    %% CV Flow
    Botnet_DS --> S3 --> Tr3 --> P3 --> N6 --> T3 --> Pr3 --> E3 --> N7

    %% Click Events - Centralized MNIST
    click R1 "https://github.com/nicola-cecere/massively-parallel-ml/blob/main/MNIST/centralized_version/read.py"
    click P1 "https://github.com/nicola-cecere/massively-parallel-ml/blob/main/MNIST/centralized_version/assignCluster.py"
    click T1 "https://github.com/nicola-cecere/massively-parallel-ml/blob/main/MNIST/centralized_version/kmeans.py"
    click N1 "https://github.com/nicola-cecere/massively-parallel-ml/blob/main/MNIST/centralized_version/mnist_centralized.ipynb"

    %% Click Events - Centralized Botnet
    click R2 "https://github.com/nicola-cecere/massively-parallel-ml/blob/main/botnet/centralized_version/preprocess.py"
    click N2 "https://github.com/nicola-cecere/massively-parallel-ml/blob/main/botnet/centralized_version/normalize.py"
    click T2 "https://github.com/nicola-cecere/massively-parallel-ml/blob/main/botnet/centralized_version/train.py"
    click Pr2 "https://github.com/nicola-cecere/massively-parallel-ml/blob/main/botnet/centralized_version/predict.py"
    click E2 "https://github.com/nicola-cecere/massively-parallel-ml/blob/main/botnet/centralized_version/accuracy.py"
    click N3 "https://github.com/nicola-cecere/massively-parallel-ml/blob/main/botnet/centralized_version/botnet_centralized.ipynb"

    %% Click Events - Parallel
    click SD1 "https://github.com/nicola-cecere/massively-parallel-ml/blob/main/MNIST/parallel_version/mnist_parallelized.ipynb"
    click SD2 "https://github.com/nicola-cecere/massively-parallel-ml/blob/main/botnet/parallel_version/main.py"
    click Exec2 "https://github.com/nicola-cecere/massively-parallel-ml/blob/main/botnet/parallel_version/preprocess.py"
    click Col2 "https://github.com/nicola-cecere/massively-parallel-ml/blob/main/botnet/parallel_version/normalize.py"
    click Col2 "https://github.com/nicola-cecere/massively-parallel-ml/blob/main/botnet/parallel_version/train.py"
    click Col2 "https://github.com/nicola-cecere/massively-parallel-ml/blob/main/botnet/parallel_version/predict.py"
    click Col2 "https://github.com/nicola-cecere/massively-parallel-ml/blob/main/botnet/parallel_version/accuracy.py"
    click N5 "https://github.com/nicola-cecere/massively-parallel-ml/blob/main/botnet/parallel_version/botnet_parallelized.ipynb"

    %% Click Events - Cross-Validation
    click S3 "https://github.com/nicola-cecere/massively-parallel-ml/blob/main/botnet/cross_validation/get_block_data.py"
    click Tr3 "https://github.com/nicola-cecere/massively-parallel-ml/blob/main/botnet/cross_validation/transform.py"
    click P3 "https://github.com/nicola-cecere/massively-parallel-ml/blob/main/botnet/cross_validation/preprocess.py"
    click N6 "https://github.com/nicola-cecere/massively-parallel-ml/blob/main/botnet/cross_validation/normalize.py"
    click T3 "https://github.com/nicola-cecere/massively-parallel-ml/blob/main/botnet/cross_validation/train.py"
    click Pr3 "https://github.com/nicola-cecere/massively-parallel-ml/blob/main/botnet/cross_validation/predict.py"
    click E3 "https://github.com/nicola-cecere/massively-parallel-ml/blob/main/botnet/cross_validation/accuracy.py"
    click N7 "https://github.com/nicola-cecere/massively-parallel-ml/blob/main/botnet/cross_validation/botnet_cross.ipynb"

    %% Packaging
    subgraph "Packaging & Config"
        SP["setup.py"]:::pythonModule
        PJ["pyproject.toml"]:::pythonModule
    end

    %% Styles
    classDef pythonModule fill:#BBDEFB,stroke:#333,stroke-width:1px
    classDef spark fill:#FFE0B2,stroke:#333,stroke-width:1px
    classDef notebook fill:#C8E6C9,stroke:#333,stroke-width:1px
    classDef datasource fill:#FFCCBC,stroke:#333,stroke-width:1px
```
