How manishghoshal99/ccc_a1 works
Project Type and Purpose
• This project is a data analytics pipeline designed to process large Mastodon NDJSON datasets. It leverages MPI for parallel processing along with memory-mapped I/O, multiprocessing, and optional Numba acceleration.
• Its goal is to efficiently preprocess and analyze massive social media data sets (even up to 144 GB) and output sentiment reports and statistical summaries.
File Structure
• The top-level directories are clearly separated:
• data – holds input NDJSON files.
• docs – contains documentation (including an assignment-related PDF).
• output and output_dir – store logs, results, and outputs from various processing runs.
• src – contains all code modules:
∙ main.py: The MPI-enabled main driver script.
∙ config.py, util.py, MastodonData.py: Utilities and configuration.
∙ data_preprocessor.py: Handles optional preprocessing of data.
∙ analysis.py: A sequential (non-MPI) analysis option.
• tests – houses test files for data processing and sentiment analysis.
• slurm – includes batch job scripts for different node/core configurations relevant for HPC environments.
• requirements.txt and venv help with dependency management.
• The presence of MPI instructions in the README and Slurm scripts indicates that parallel computation and HPC integration are central to the architecture.
Key Architectural Components
• Data Source: NDJSON files in the data directory.
• Preprocessing Module: Representing the process run by data_preprocessor.py.
• Main Processing Module: The MPI-based processing driven by src/main.py.
• Sequential Analysis Module: Provided by analysis.py for non-parallel running.
• Configuration and Utilities: The config.py and util.py act as helper modules.
• Logging & Output: Two separate components – output (for logs/results) and output_dir (for larger cascade and sapphire outputs).
• Job Management: The slurm directory indicates integration with HPC job schedulers such as SLURM.
• Testing: While not an operational component, the tests directory indicates that there is unit/integration testing for ensuring correctness.
Relationships and Interactions
• Data flows from the input files (data/) into preprocessing (if that step is invoked) and/or directly into the main analysis.
• The main processing application (src/main.py) is executed in parallel using MPI; it interacts with configuration and utility modules to process data.
• Processed data and logs are written to the output directories.
• There is an alternative sequential analysis path using analysis.py which processes similar input data.
• The entire processing pipeline can be orchestrated via SLURM scripts (in the slurm directory) to run on HPC clusters.
Important Architectural Patterns and Technologies
• Parallel Processing with MPI and multiprocessing.
• Performance optimization through memory-mapped I/O and optional JIT compilation (Numba).
• Clear separation of concerns: input data, preprocessing, core analysis, and output.
• Use of a configuration module to manage paths and possibly environment-specific settings.
• Test-driven development model (as seen in the tests directory).