How turjo7/parallel_computing_sentiment_analysis works
Project type and purpose
• This is a distributed batch-processing pipeline for sentiment analysis, built on an HTCondor cluster with NFS-backed shared storage.
• It orchestrates a DAG of jobs (data prep, Python-based classification, aggregation) to process input.txt and produce sentiment_results.csv.
File structure
• Top-level:
• HTCondor files (.dag, .sub, .out, .err, rescue files) – define and track DAG jobs.
• Shell scripts (agg_to_csv.sh, data.sh, diagnostic.sh) – wrappers for data and aggregation steps.
• Python script (data.py) – likely does preprocessing or classification.
• Configuration and log directories (error/, outputs/).
• Raw inputs (input.txt) and final output (sentiment_results.csv).
Extract architecture from README
• Components: NFS server, NFS clients (HTCondor execute nodes), Condor submit/manager node.
• Setup: UID_DOMAIN alignment, NFS export/mount, Python venv for transformers+torch.
• Execution: condor_submit_dag runs sentiment.dag, jobs execute on workers, share data via NFS.
a. Main components
• NFS Server (exports /home/ubuntu/sentiment)
• HTCondor Submit/Manager Node (runs condor_manage, condor_submit_dag)
• HTCondor Execute Nodes (multiple clients mounting NFS)
• Shared storage (NFS mount) containing code, scripts, input.txt, and DAG files
• Python runtime (venv with transformers, torch)
• External user interface (SSH or CLI on submit node)
b. Relationships and data flow
1. Admin sets up NFS export on server → clients mount directory.
2. Submit node uses shared NFS for HTCondor config (UID_DOMAIN) and job files.
3. User invokes condor_submit_dag sentiment.dag on submit node.
4. DAG manager dispatches jobs to execute nodes via Condor negotiator.
5. Each job reads/writes from NFS share:
• data.sh/data.py preprocess input.txt → intermediate outputs in outputs/
• sentiment*.sub runs Python classifier on chunks
• agg_to_csv.sh aggregates results into sentiment_results.csv
6. Logs and errors also reside on NFS in error/ and outputs/.
c. Architectural patterns and principles
• Shared-nothing compute model with a central NFS store for code and data (minimize duplication).
• DAG orchestration for clear dependency management.
• Modularity via separate scripts (preprocess, classify, aggregate).
d. Key technologies/frameworks
• HTCondor (DAGMan, condor_submit_dag, .sub files)
• NFS (exports, mounts)
• Python3 venv with Transformers and Torch
• Bash for orchestration scripts