How ankitkr8540/pylinguist-paper works
Project type and purpose
• PyLinguist is a CLI‐driven data‐processing and translation pipeline (not a web app). It automates translation of Python code into multiple target languages via a two‐stage process: basic keyword mapping + Google Translate (Stage 1), then contextual refinement using LLMs (Stage 2).
• The goal is accessibility: preserve code functionality while rendering comments, identifiers, and keywords into the target language.
File structure
• Top‐level:
• scripts/translate.py, scripts/evaluate.py: CLI entry points
• data/: raw and preprocessed datasets
• pylinguist/: core library
• Visualization.ipynb, visualizations/: charting
• Results/: output files for every language/model combination
• pylinguist/models/:
• stage1/: base + Google translation implementation
• stage2/: base + GPT + Claude implementations
• pylinguist/utils/: preprocessor, logger, partial translator, language extractor
Key insights from README
• Two‐stage pipeline invoked via translate.py
• stage1 uses keyword mapping + Google Translate API
• stage2 uses OpenAI GPT or Anthropic Claude API
• Evaluation module computes metrics, results stored under Results/
• Visualization notebook consumes those results
Main components
a. CLI Interface (“scripts/translate.py” & “scripts/evaluate.py”)
b. Data Preprocessor (pylinguist/utils/preprocessor)
c. Stage 1 Translator
• Keyword Mapper (Unipy_Keywords.csv)
• Google Translate API (deep-translator under the hood)
d. Stage 2 Refiner
• GPT Model (OpenAI API)
• Claude Model (Anthropic API)
e. Evaluation Engine (scripts/evaluate.py + pylinguist/evaluation)
f. Data Stores
• Input: HuggingFace Python code dataset (data/input)
• Keywords: data/keywords
• Outputs: Results/… CSV & JSON
g. Visualization (Visualization.ipynb + visualizations/)
Relationships and data flow
• User runs CLI → Data Preprocessor loads dataset → Stage 1 Translator emits partial translations → Stage 2 Refiner takes Stage 1 outputs → Final translated code outputs to Results/
• Evaluation Engine reads both original + translated code, runs metrics → writes evaluation_summary_*.json and details_*.csv to Results/
• Visualization notebook reads Results/ files to produce charts.
Architectural patterns and principles
• Modular pipeline: each stage is a separate component
• Strategy pattern: interchangeable translation engines (Google vs. keyword mapping; GPT vs. Claude)
• Separation of concerns: utils, models, evaluation, scripts
Technologies and external services
• Python CLI (argparse)
• pandas, numpy for data handling
• deep-translator for Google Translate
• openai, anthropic for Stage 2 LLMs
• HuggingFace datasets
• Jupyter for visualization