# moalthaf/paraphrasingproject architecture

> Architecture diagram of the GitHub repository moalthaf/paraphrasingproject, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-08-21). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/moalthaf/paraphrasingproject
- Repository: https://github.com/moalthaf/paraphrasingproject

## Overview

##### Project Type and Purpose
- This is a data-processing and model-inference pipeline (“ParaphrasingProject”) that takes SQL queries, generates paraphrases using multiple LLM backends, evaluates them, and produces structured result files for analysis.

##### High-Level Components
a. Data Layer
- Raw/interim CSVs under `data/`
- Processed output CSVs
b. Preprocessing Module
- `src/prepare_dataset.py` for dataset cleaning and formatting
- Utilities in `src/utils/sql_utils.py`
c. Model Inference Layer
- `models/llama.py`, `models/mistral.py`, `models/qwen.py`
- Shared prompt templates in `models/prompt_templates.py`
d. Evaluation Module
- `src/utils/paraphrase_score.py` computes quality metrics
- Logging via `src/utils/logger.py`
e. Orchestration Script
- `main.py` calls preprocessing → model wrappers → evaluation → writes to `result/`
- `parallel.sh` to launch batch runs in parallel
f. Analysis Notebook
- `notebook/result_analysis.ipynb` for manual/visual inspection of `result/`

##### Component Relationships and Data Flows
- Data Layer → Preprocessing Module
- Preprocessing → Orchestration Script (`main.py`)
- Orchestration → Model Inference Layer
- Model outputs → Evaluation Module
- Evaluation results → Result Storage (`result/`)
- Final results → Analysis Notebook

##### Architectural Patterns and Principles
- Pipeline/ETL pattern: clear separation of extract (data), transform (preprocess & score), load (write results)
- Modular design: each model backend and utility lives in its own module for easy extension
- Script-based orchestration: `main.py` as single entry point; `parallel.sh` for scalability

##### Technologies and Frameworks
- Python 3.x
- Pandas/CSV I/O
- LLM backends (wrapped in `models/*.py`)
- Shell scripting for parallel runs
- Jupyter for exploratory analysis

2. Preprocessing module (`prepare_dataset.py`) between Raw and Processed.
6. Evaluation to Results store.
7. Off to the side, show Notebook consuming the Results for analysis.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data Stores
    subgraph "Data Stores"
        Raw["Raw Data\n(data/interim)"]:::dataStore
        Processed["Processed Data\n(data/processed)"]:::dataStore
        Results["Results\n(result/)"]:::dataStore
    end

    %% Preprocessing Module
    subgraph "Preprocessing"
        Prep["prepare_dataset.py"]:::processing
        SQL["sql_utils.py"]:::processing
    end

    %% Orchestration
    subgraph "Orchestration"
        Main["main.py"]:::orchestration
        Parallel["parallel.sh"]:::orchestration
    end

    %% Model Inference Layer
    subgraph "Model Inference"
        Llama["llama.py"]:::model
        Mistral["mistral.py"]:::model
        Qwen["qwen.py"]:::model
        Prompts["prompt_templates.py"]:::model
    end

    %% Evaluation Module
    subgraph "Evaluation"
        Score["paraphrase_score.py"]:::evaluation
        Log["logger.py"]:::evaluation
    end

    %% Analysis
    Notebook["result_analysis.ipynb"]:::analysis

    %% Data Flow
    Raw -->|"clean & format"| Prep
    SQL --> Prep
    Prep -->|"cleaned dataset"| Processed
    Processed -->|"load dataset"| Main
    Parallel --> Main
    Main -->|"generate paraphrases"| Llama
    Main -->|"generate paraphrases"| Mistral
    Main -->|"generate paraphrases"| Qwen
    Prompts --> Llama
    Prompts --> Mistral
    Prompts --> Qwen
    Llama -->|"paraphrase outputs"| Score
    Mistral --> Score
    Qwen --> Score
    Score -->|"score paraphrases"| Log
    Score -->|"scored data"| Results
    Log --> Results
    Results -->|"analyze results"| Notebook

    %% Click Events
    click Raw "https://github.com/moalthaf/paraphrasingproject/blob/main/data/interim/generated_queries.csv"
    click Processed "https://github.com/moalthaf/paraphrasingproject/blob/main/data/processed/output_paraphrased.csv"
    click Prep "https://github.com/moalthaf/paraphrasingproject/blob/main/src/prepare_dataset.py"
    click SQL "https://github.com/moalthaf/paraphrasingproject/blob/main/src/utils/sql_utils.py"
    click Llama "https://github.com/moalthaf/paraphrasingproject/blob/main/models/llama.py"
    click Mistral "https://github.com/moalthaf/paraphrasingproject/blob/main/models/mistral.py"
    click Qwen "https://github.com/moalthaf/paraphrasingproject/blob/main/models/qwen.py"
    click Prompts "https://github.com/moalthaf/paraphrasingproject/blob/main/models/prompt_templates.py"
    click Score "https://github.com/moalthaf/paraphrasingproject/blob/main/src/utils/paraphrase_score.py"
    click Log "https://github.com/moalthaf/paraphrasingproject/blob/main/src/utils/logger.py"
    click Main "https://github.com/moalthaf/paraphrasingproject/blob/main/main.py"
    click Parallel "https://github.com/moalthaf/paraphrasingproject/blob/main/parallel.sh"
    click Results "https://github.com/moalthaf/paraphrasingproject/tree/main/result/"
    click Notebook "https://github.com/moalthaf/paraphrasingproject/blob/main/notebook/result_analysis.ipynb"

    %% Styles
    classDef dataStore fill:#cce5ff,stroke:#004085,color:#004085,stroke-width:1px;
    classDef processing fill:#d4edda,stroke:#155724,color:#155724,stroke-width:1px;
    classDef orchestration fill:#fff3cd,stroke:#856404,color:#856404,stroke-width:1px;
    classDef model fill:#ffe5b4,stroke:#cc7a00,color:#cc7a00,stroke-width:1px;
    classDef evaluation fill:#e2d6f3,stroke:#5a3199,color:#5a3199,stroke-width:1px;
    classDef analysis fill:#f3e6fa,stroke:#6f42c1,color:#6f42c1,stroke-width:1px;
```
