How moalthaf/paraphrasingproject works
Project Type and Purpose
• This is a data-processing and model-inference pipeline (“ParaphrasingProject”) that takes SQL queries, generates paraphrases using multiple LLM backends, evaluates them, and produces structured result files for analysis.
High-Level Components
a. Data Layer
• Raw/interim CSVs under data/
• Processed output CSVs
b. Preprocessing Module
• src/prepare_dataset.py for dataset cleaning and formatting
• Utilities in src/utils/sql_utils.py
c. Model Inference Layer
• models/llama.py, models/mistral.py, models/qwen.py
• Shared prompt templates in models/prompt_templates.py
d. Evaluation Module
• src/utils/paraphrase_score.py computes quality metrics
• Logging via src/utils/logger.py
e. Orchestration Script
• main.py calls preprocessing → model wrappers → evaluation → writes to result/
• parallel.sh to launch batch runs in parallel
f. Analysis Notebook
• notebook/result_analysis.ipynb for manual/visual inspection of result/
Component Relationships and Data Flows
• Data Layer → Preprocessing Module
• Preprocessing → Orchestration Script (main.py)
• Orchestration → Model Inference Layer
• Model outputs → Evaluation Module
• Evaluation results → Result Storage (result/)
• Final results → Analysis Notebook
Architectural Patterns and Principles
• Pipeline/ETL pattern: clear separation of extract (data), transform (preprocess & score), load (write results)
• Modular design: each model backend and utility lives in its own module for easy extension
• Script-based orchestration: main.py as single entry point; parallel.sh for scalability
Technologies and Frameworks
• Python 3.x
• Pandas/CSV I/O
• LLM backends (wrapped in models/*.py)
• Shell scripting for parallel runs
• Jupyter for exploratory analysis
2. Preprocessing module (prepare_dataset.py) between Raw and Processed.
6. Evaluation to Results store.
7. Off to the side, show Notebook consuming the Results for analysis.