How nicola-cecere/massively-parallel-ml works
Project Type & Purpose
• This is a Python+Spark research/demo codebase illustrating “massively parallel” implementations of two core ML algorithms:
– K-Means clustering on MNIST
– Logistic Regression for botnet detection
• It provides both centralized (single‐machine) and parallelized (Spark) variants, plus cross‐validation for the botnet project.
Key Components
a. Data Sources
– MNIST image files (raw pixel data)
– Botnet network‐flow dataset
b. Centralized Pipelines (control flow)
– Data reader/loader modules (read.py)
– Preprocessing modules (normalize.py, assignCluster.py)
– Training module (train.py / kmeans.py)
– Prediction / clustering modules (predict.py)
– Evaluation modules (accuracy.py)
– Jupyter notebooks tying it all together
c. Parallel Pipelines (Spark)
– Spark driver process (main.py)
– RDD/DataFrame creation from raw data
– Parallelized map/reduce or iterative steps for K-Means and Logistic Regression
– Spark executors that run bulk computations
– Aggregation and collection of results
d. Cross‐Validation (Botnet only)
– Data split and fold generation (get_block_data.py, transform.py)
– Training/evaluation per fold
– Aggregation of fold‐metrics
e. Supporting Artifacts
– setup.py / pyproject.toml for environment & packaging
– PDF reports & slides for documentation
Interactions & Data Flow
• Raw Data → Preprocessor → Training → Model Parameters → Predictor → Evaluator → Metric outputs
• In parallel mode, the “Training” and “Prediction” stages run distributed across Spark executors; the driver orchestrates.
• In cross‐validation, multiple parallel “fold” pipelines run sequentially or in parallel depending on your Spark setup.
Architectural Patterns & Principles
• Pipeline pattern: clear stages (ingest → preprocess → train → predict → eval)
• Separation of concerns: centralized vs. parallel code under distinct directories
• Master–worker model in Spark: one driver, many executors
• Modularity: each algorithm (K-Means, Logistic Regression) has its own set of modules