# manishghoshal99/ccc_a1 architecture

> Architecture diagram of the GitHub repository manishghoshal99/ccc_a1, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-07). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/manishghoshal99/ccc_a1
- Repository: https://github.com/manishghoshal99/ccc_a1

## Overview

##### Project Type and Purpose
- This project is a data analytics pipeline designed to process large Mastodon NDJSON datasets. It leverages MPI for parallel processing along with memory-mapped I/O, multiprocessing, and optional Numba acceleration.
- Its goal is to efficiently preprocess and analyze massive social media data sets (even up to 144 GB) and output sentiment reports and statistical summaries.

##### File Structure
- The top-level directories are clearly separated:
- data – holds input NDJSON files.
- docs – contains documentation (including an assignment-related PDF).
- output and output_dir – store logs, results, and outputs from various processing runs.
- src – contains all code modules:
∙ main.py: The MPI-enabled main driver script.
∙ config.py, util.py, MastodonData.py: Utilities and configuration.
∙ data_preprocessor.py: Handles optional preprocessing of data.
∙ analysis.py: A sequential (non-MPI) analysis option.
- tests – houses test files for data processing and sentiment analysis.
- slurm – includes batch job scripts for different node/core configurations relevant for HPC environments.
- requirements.txt and venv help with dependency management.
- The presence of MPI instructions in the README and Slurm scripts indicates that parallel computation and HPC integration are central to the architecture.

##### Key Architectural Components
- Data Source: NDJSON files in the data directory.
- Preprocessing Module: Representing the process run by data_preprocessor.py.
- Main Processing Module: The MPI-based processing driven by src/main.py.
- Sequential Analysis Module: Provided by analysis.py for non-parallel running.
- Configuration and Utilities: The config.py and util.py act as helper modules.
- Logging & Output: Two separate components – output (for logs/results) and output_dir (for larger cascade and sapphire outputs).
- Job Management: The slurm directory indicates integration with HPC job schedulers such as SLURM.
- Testing: While not an operational component, the tests directory indicates that there is unit/integration testing for ensuring correctness.

##### Relationships and Interactions
- Data flows from the input files (data/) into preprocessing (if that step is invoked) and/or directly into the main analysis.
- The main processing application (src/main.py) is executed in parallel using MPI; it interacts with configuration and utility modules to process data.
- Processed data and logs are written to the output directories.
- There is an alternative sequential analysis path using analysis.py which processes similar input data.
- The entire processing pipeline can be orchestrated via SLURM scripts (in the slurm directory) to run on HPC clusters.

##### Important Architectural Patterns and Technologies
- Parallel Processing with MPI and multiprocessing.
- Performance optimization through memory-mapped I/O and optional JIT compilation (Numba).
- Clear separation of concerns: input data, preprocessing, core analysis, and output.
- Use of a configuration module to manage paths and possibly environment-specific settings.
- Test-driven development model (as seen in the tests directory).

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data Layer
    subgraph "Data Layer"
        DS["Data Sources (NDJSON)"]:::dataColor
    end

    %% Preprocessing
    subgraph "Preprocessing"
        PRE["Preprocessing: data_preprocessor.py"]:::processColor
    end

    %% Processing Modules
    subgraph "Processing Modules"
        MPI["MPI Main Processing: main.py"]:::processColor
        SA["Sequential Analysis: analysis.py"]:::processColor
    end

    %% Helpers
    subgraph "Helpers"
        CONFIG["Configuration: config.py"]:::helperColor
        UTIL["Utilities: util.py & MastodonData.py"]:::helperColor
    end

    %% Output & Logging
    subgraph "Output & Logging"
        OUT["Output (logs/results)"]:::outputColor
        ODIR["Output Dir (cascade outputs)"]:::outputColor
    end

    %% HPC Orchestration
    subgraph "HPC Orchestration"
        HPC["HPC Job Scheduling (SLURM)"]:::jobColor
    end

    %% Support
    subgraph "Support"
        TEST["Testing"]:::supportColor
        DOCS["Documentation"]:::supportColor
    end

    %% Data Flow
    DS -->|"input"| PRE
    DS -->|"raw_data"| MPI
    DS -->|"raw_data"| SA
    PRE -->|"processed_data"| MPI
    PRE -->|"processed_data"| SA

    %% Helpers to Processing
    CONFIG -->|"config"| MPI
    CONFIG -->|"config"| SA
    UTIL -->|"helpers"| MPI
    UTIL -->|"helpers"| SA

    %% HPC scheduling to MPI
    HPC -->|"schedule"| MPI

    %% Processing to Output
    MPI -->|"results"| OUT
    MPI -->|"results"| ODIR
    SA -->|"results"| OUT
    SA -->|"results"| ODIR

    %% Click Events
    click DS "https://github.com/manishghoshal99/ccc_a1/tree/main/data/"
    click PRE "https://github.com/manishghoshal99/ccc_a1/blob/main/src/data_preprocessor.py"
    click MPI "https://github.com/manishghoshal99/ccc_a1/blob/main/src/main.py"
    click SA "https://github.com/manishghoshal99/ccc_a1/blob/main/src/analysis.py"
    click CONFIG "https://github.com/manishghoshal99/ccc_a1/blob/main/src/config.py"
    click UTIL "https://github.com/manishghoshal99/ccc_a1/blob/main/src/util.py"
    click OUT "https://github.com/manishghoshal99/ccc_a1/tree/main/output/"
    click ODIR "https://github.com/manishghoshal99/ccc_a1/tree/main/output_dir/"
    click HPC "https://github.com/manishghoshal99/ccc_a1/tree/main/slurm/"
    click TEST "https://github.com/manishghoshal99/ccc_a1/tree/main/tests/"
    click DOCS "https://github.com/manishghoshal99/ccc_a1/tree/main/docs/"

    %% Styles
    classDef dataColor fill:#ffcc00,stroke:#333,stroke-width:2px;
    classDef processColor fill:#87CEEB,stroke:#333,stroke-width:2px;
    classDef helperColor fill:#98fb98,stroke:#333,stroke-width:2px;
    classDef outputColor fill:#ffb6c1,stroke:#333,stroke-width:2px;
    classDef jobColor fill:#d3d3d3,stroke:#333,stroke-width:2px;
    classDef supportColor fill:#dda0dd,stroke:#333,stroke-width:2px;
```
