# turjo7/parallel_computing_sentiment_analysis architecture

> Architecture diagram of the GitHub repository turjo7/parallel_computing_sentiment_analysis, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-21). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/turjo7/parallel_computing_sentiment_analysis
- Repository: https://github.com/turjo7/parallel_computing_sentiment_analysis

## Overview

##### Project type and purpose
- This is a distributed batch-processing pipeline for sentiment analysis, built on an HTCondor cluster with NFS-backed shared storage.
- It orchestrates a DAG of jobs (data prep, Python-based classification, aggregation) to process `input.txt` and produce `sentiment_results.csv`.

##### File structure
- Top-level:
- HTCondor files (.dag, .sub, .out, .err, rescue files) – define and track DAG jobs.
- Shell scripts (agg_to_csv.sh, data.sh, diagnostic.sh) – wrappers for data and aggregation steps.
- Python script (data.py) – likely does preprocessing or classification.
- Configuration and log directories (error/, outputs/).
- Raw inputs (input.txt) and final output (sentiment_results.csv).

##### Extract architecture from README
- Components: NFS server, NFS clients (HTCondor execute nodes), Condor submit/manager node.
- Setup: UID_DOMAIN alignment, NFS export/mount, Python venv for transformers+torch.
- Execution: condor_submit_dag runs `sentiment.dag`, jobs execute on workers, share data via NFS.

a. Main components
- NFS Server (exports /home/ubuntu/sentiment)
- HTCondor Submit/Manager Node (runs condor_manage, condor_submit_dag)
- HTCondor Execute Nodes (multiple clients mounting NFS)
- Shared storage (NFS mount) containing code, scripts, input.txt, and DAG files
- Python runtime (venv with transformers, torch)
- External user interface (SSH or CLI on submit node)
b. Relationships and data flow
1. Admin sets up NFS export on server → clients mount directory.
2. Submit node uses shared NFS for HTCondor config (UID_DOMAIN) and job files.
3. User invokes `condor_submit_dag sentiment.dag` on submit node.
4. DAG manager dispatches jobs to execute nodes via Condor negotiator.
5. Each job reads/writes from NFS share:
- data.sh/data.py preprocess input.txt → intermediate outputs in outputs/
- sentiment*.sub runs Python classifier on chunks
- agg_to_csv.sh aggregates results into sentiment_results.csv
6. Logs and errors also reside on NFS in `error/` and `outputs/`.
c. Architectural patterns and principles
- Shared-nothing compute model with a central NFS store for code and data (minimize duplication).
- DAG orchestration for clear dependency management.
- Modularity via separate scripts (preprocess, classify, aggregate).
d. Key technologies/frameworks
- HTCondor (DAGMan, condor_submit_dag, .sub files)
- NFS (exports, mounts)
- Python3 venv with Transformers and Torch
- Bash for orchestration scripts

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Storage Layer
    subgraph "Storage Layer"
        direction TB
        NFS_Server["NFS Server"]:::storage
        Shared_Storage["Shared Storage\n(/home/ubuntu/sentiment)"]:::storage
    end

    NFS_Server -->|export NFS| Shared_Storage

    %% Control Layer
    subgraph "Control Layer"
        direction TB
        Submit_Node["HTCondor Submit/Manager Node\n(UID_DOMAIN configured)"]:::control
        Condor_Scheduler["Condor Scheduler\n(DAGMan & Negotiator)"]:::control
    end

    Shared_Storage -->|mount NFS| Submit_Node
    Submit_Node -->|condor_submit_dag sentiment.dag| Condor_Scheduler

    %% Compute Layer
    subgraph "Compute Layer"
        direction TB
        Execute_Nodes["HTCondor Execute Nodes\n(mount NFS & run jobs)"]:::compute
    end

    Condor_Scheduler -->|dispatch jobs| Execute_Nodes
    Execute_Nodes -->|read/write files| Shared_Storage

    %% DAG Subgraph
    subgraph "DAG: sentiment.dag" 
        direction TB
        Data_Prep["data.sh / data.py"]:::script
        Classification["sentiment_*.sub\n(4 jobs)"]:::script
        Aggregation["agg_to_csv.sh / agg.sub"]:::script

        Data_Prep -->|on success| Classification
        Classification -->|on success| Aggregation
    end

    Submit_Node -->|manage DAG| Data_Prep
    Execute_Nodes --> Data_Prep
    Execute_Nodes --> Classification
    Execute_Nodes --> Aggregation

    %% Files on Shared Storage
    Input_File["input.txt"]:::storage
    Output_File["sentiment_results.csv"]:::storage
    Diagnostics["diagnostic.sh"]:::script
    README["readme.md"]:::script
    Gitignore[".gitignore"]:::script

    Shared_Storage --> Input_File
    Shared_Storage --> Diagnostics
    Shared_Storage --> README
    Shared_Storage --> Gitignore
    Aggregation -->|writes| Output_File

    %% Click Events
    click sentiment.dag "https://github.com/turjo7/parallel_computing_sentiment_analysis/blob/main/sentiment.dag"
    click Data_Prep "https://github.com/turjo7/parallel_computing_sentiment_analysis/blob/main/data.sh"
    click Data_Prep "https://github.com/turjo7/parallel_computing_sentiment_analysis/blob/main/data.py"
    click Data_Prep "https://github.com/turjo7/parallel_computing_sentiment_analysis/blob/main/data.sub"
    click Classification "https://github.com/turjo7/parallel_computing_sentiment_analysis/blob/main/agg_to_csv.sh"
    click Classification "https://github.com/turjo7/parallel_computing_sentiment_analysis/blob/main/agg.sub"
    click Input_File "https://github.com/turjo7/parallel_computing_sentiment_analysis/blob/main/input.txt"
    click Output_File "https://github.com/turjo7/parallel_computing_sentiment_analysis/blob/main/sentiment_results.csv"
    click Diagnostics "https://github.com/turjo7/parallel_computing_sentiment_analysis/blob/main/diagnostic.sh"
    click README "https://github.com/turjo7/parallel_computing_sentiment_analysis/blob/main/readme.md"
    click Gitignore "https://github.com/turjo7/parallel_computing_sentiment_analysis/blob/main/.gitignore"

    %% Styles
    classDef storage fill:#ADD8E6,stroke:#000,stroke-width:1px
    classDef control fill:#90EE90,stroke:#000,stroke-width:1px
    classDef compute fill:#FFA500,stroke:#000,stroke-width:1px
    classDef script fill:#D3D3D3,stroke:#000,stroke-width:1px
```
