How com6012/scalableml works
• Code – contains Python scripts (e.g., LogMining100.py, LogMiningBig.py) that serve as practical examples or lab exercises.
• Data – provides various sample datasets (e.g., Advertising.csv, MovieLens100k.data, NASA logs, iris.csv, spambase.data, etc.) used in practical sessions.
• HPC – includes scripts for running batch jobs (e.g., sample_batch.sh) on HPC clusters.
• Lab notes – Markdown files (e.g., Lab 1 – Introduction to Spark and HPC.md up through Lab 8, etc.) that describe and guide through different aspects of scalable machine learning using Spark.
• Output – holds sample results from running labs.
• Slides – provides lecture slides and overviews of the modules.
• README – contextualizes the project as a part of a course curriculum (Spring 2025) with introductory material, session breakdown, and acknowledgements.
Main Components
a. Educational Modules Layer:
– Lab session files (the Markdown files with instructions for each session)
– Lecture slides
b. Code Component:
– Python scripts in the Code directory that implement practical examples (e.g., log mining operations with Spark).
c. Data Repository:
– Various datasets used for exercises (advertising, movie ratings, NASA logs, etc.).
d. Execution Platform:
– HPC cluster (illustrated through the HPC directory and sample batch scripts) where students access resources via SSH/VPN and run Spark jobs.
e. Processing Layer:
– Apache Spark (PySpark 3.5.4 as specified) that underpins the scalable machine learning operations.
f. Output/Results:
– Output directory that logs results and outputs from the labs.
Relationships and Interactions
– The “User” (student or instructor) connects to the HPC cluster (using SSH with MFA) to submit jobs.
– Upon submission, the HPC system schedules Spark jobs defined in the Code scripts.
– Apache Spark, running on the cluster, processes the datasets (from the Data folder) using distributed computations.
– The processed results are stored in the Output folder.
– Lab notes and slides provide instructions and context throughout the process.
Architectural Patterns and Design Principles
– Modular design: Separation between teaching material (labs and slides), real code (Python scripts), data, and the execution environment (HPC + Spark).
– Client–Server model: Students (clients) interact via SSH/VPN with the remote HPC cluster (server) to run distributed Spark jobs.
– Educational orchestration: The flow from instructions (lab notes) to execution (Spark jobs on HPC) to result analysis forms a tightly integrated learning pipeline.
Technologies and Tools to Highlight
– Apache Spark (specifically PySpark 3.5.4) for distributed processing.
– HPC Cluster environment and accompanying shell scripts for job management.
– Python for the code developed in labs.
– Standard datasets that feed into machine learning exercises.
– Clear labels for each component (e.g., “Lab Sessions”, “Python Code”, “Datasets”, “HPC Cluster”, “Apache Spark Processing”, “Output Results”).
• From “User” to “HPC” (for job submission),
• From “HPC” to “Spark” (job processing across the cluster),
– Annotated connection lines where HPC scripts in the HPC directory trigger the Spark jobs, and where the lab instructions tie the components together in a learning pipeline.