# ankitkr8540/pylinguist-paper architecture

> Architecture diagram of the GitHub repository ankitkr8540/pylinguist-paper, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-20). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/ankitkr8540/pylinguist-paper
- Repository: https://github.com/ankitkr8540/pylinguist-paper

## Overview

##### Project type and purpose
- PyLinguist is a CLI‐driven data‐processing and translation pipeline (not a web app). It automates translation of Python code into multiple target languages via a two‐stage process: basic keyword mapping + Google Translate (Stage 1), then contextual refinement using LLMs (Stage 2).
- The goal is accessibility: preserve code functionality while rendering comments, identifiers, and keywords into the target language.

##### File structure
- Top‐level:
- scripts/translate.py, scripts/evaluate.py: CLI entry points
- data/: raw and preprocessed datasets
- pylinguist/: core library
- Visualization.ipynb, visualizations/: charting
- Results/: output files for every language/model combination
- pylinguist/models/:
- stage1/: base + Google translation implementation
- stage2/: base + GPT + Claude implementations
- pylinguist/utils/: preprocessor, logger, partial translator, language extractor

##### Key insights from README
- Two‐stage pipeline invoked via translate.py
- stage1 uses keyword mapping + Google Translate API
- stage2 uses OpenAI GPT or Anthropic Claude API
- Evaluation module computes metrics, results stored under Results/
- Visualization notebook consumes those results

##### Main components
a. CLI Interface (“scripts/translate.py” & “scripts/evaluate.py”)
b. Data Preprocessor (pylinguist/utils/preprocessor)
c. Stage 1 Translator
- Keyword Mapper (Unipy_Keywords.csv)
- Google Translate API (deep-translator under the hood)
d. Stage 2 Refiner
- GPT Model (OpenAI API)
- Claude Model (Anthropic API)
e. Evaluation Engine (scripts/evaluate.py + pylinguist/evaluation)
f. Data Stores
- Input: HuggingFace Python code dataset (data/input)
- Keywords: data/keywords
- Outputs: Results/… CSV & JSON
g. Visualization (Visualization.ipynb + visualizations/)

##### Relationships and data flow
- User runs CLI → Data Preprocessor loads dataset → Stage 1 Translator emits partial translations → Stage 2 Refiner takes Stage 1 outputs → Final translated code outputs to Results/
- Evaluation Engine reads both original + translated code, runs metrics → writes evaluation_summary_*.json and details_*.csv to Results/
- Visualization notebook reads Results/ files to produce charts.

##### Architectural patterns and principles
- Modular pipeline: each stage is a separate component
- Strategy pattern: interchangeable translation engines (Google vs. keyword mapping; GPT vs. Claude)
- Separation of concerns: utils, models, evaluation, scripts

##### Technologies and external services
- Python CLI (argparse)
- pandas, numpy for data handling
- deep-translator for Google Translate
- openai, anthropic for Stage 2 LLMs
- HuggingFace datasets
- Jupyter for visualization

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% CLI Layer
    subgraph "CLI Layer"
        CLI_Translate["CLI: translate.py"]:::ui
        CLI_Evaluate["CLI: evaluate.py"]:::ui
    end

    %% Preprocessing Layer
    subgraph "Preprocessing"
        Preprocessor["Data Preprocessor"]:::core
        LanguageExtractor["Language Extractor"]:::core
        PartialTranslator["Partial Translator"]:::core
        Logger["Logger Utility"]:::core
    end

    %% Stage 1 Translation
    subgraph "Stage1 Translator"
        S1Base["Stage1 Base"]:::core
        KeywordMapper["Keyword Mapper"]:::core
        GoogleImpl["Google Translate Impl"]:::core
    end

    %% Stage 2 Refinement
    subgraph "Stage2 Refiner"
        S2Base["Stage2 Base"]:::core
        GPTImpl["GPT (OpenAI) Impl"]:::core
        ClaudeImpl["Claude (Anthropic) Impl"]:::core
    end

    %% Data Stores
    InputDataset(("Input Dataset")):::dataStore
    KeywordsStore(("Keywords CSV")):::dataStore
    ResultsStore(("Results Folder")):::dataStore

    %% External Services
    GoogleAPI((Google Translate API)):::external
    OpenAIAPI((OpenAI API)):::external
    AnthropicAPI((Anthropic API)):::external

    %% Visualization
    subgraph "Visualization"
        Notebook["Jupyter Notebook"]:::ui
        VizScripts["Visualization Scripts"]:::ui
    end

    %% Evaluation Engine
    Evaluation["Evaluation Engine"]:::core

    %% Connections
    CLI_Translate -->|"run translate"| Preprocessor
    Preprocessor -->|".py code + metadata"| S1Base
    Preprocessor --> LanguageExtractor
    Preprocessor --> PartialTranslator
    Preprocessor --> Logger
    S1Base -->|"keyword mapping"| KeywordMapper
    S1Base -->|"google translate"| GoogleImpl
    KeywordMapper -->|"uses keywords"| KeywordsStore
    GoogleImpl --> GoogleAPI
    KeywordMapper --> S2Base
    GoogleImpl --> S2Base
    KeywordsStore -.-> dataLink
    S2Base -->|"refine translation"| GPTImpl
    S2Base --> GPTImpl
    S2Base --> ClaudeImpl
    GPTImpl --> OpenAIAPI
    ClaudeImpl --> AnthropicAPI
    GPTImpl -->|"final translation"| ResultsStore
    ClaudeImpl --> ResultsStore

    CLI_Evaluate -->|"run evaluate"| Evaluation
    Evaluation -->|"reads original code"| InputDataset
    Evaluation -->|"reads translations"| ResultsStore
    Evaluation -->|"writes metrics"| ResultsStore

    ResultsStore -->|"data for charts"| Notebook
    ResultsStore -->|"data for scripts"| VizScripts

    InputDataset --> Preprocessor
    KeywordsStore -.-> KeywordMapper

    %% Click Events
    click CLI_Translate "https://github.com/ankitkr8540/pylinguist-paper/blob/master/scripts/translate.py"
    click CLI_Evaluate "https://github.com/ankitkr8540/pylinguist-paper/blob/master/scripts/evaluate.py"
    click Preprocessor "https://github.com/ankitkr8540/pylinguist-paper/blob/master/data/input/preprocessor.py"
    click InputDataset "https://github.com/ankitkr8540/pylinguist-paper/blob/master/data/input/samples/python_code_dataset.csv"
    click KeywordsStore "https://github.com/ankitkr8540/pylinguist-paper/blob/master/data/keywords/Unipy_Keywords.csv"
    click S1Base "https://github.com/ankitkr8540/pylinguist-paper/blob/master/pylinguist/models/stage1/base.py"
    click GoogleImpl "https://github.com/ankitkr8540/pylinguist-paper/blob/master/pylinguist/models/stage1/google.py"
    click S2Base "https://github.com/ankitkr8540/pylinguist-paper/blob/master/pylinguist/models/stage2/base.py"
    click GPTImpl "https://github.com/ankitkr8540/pylinguist-paper/blob/master/pylinguist/models/stage2/gpt.py"
    click ClaudeImpl "https://github.com/ankitkr8540/pylinguist-paper/blob/master/pylinguist/models/stage2/claude.py"
    click LanguageExtractor "https://github.com/ankitkr8540/pylinguist-paper/blob/master/pylinguist/utils/language_extractor.py"
    click PartialTranslator "https://github.com/ankitkr8540/pylinguist-paper/blob/master/pylinguist/utils/partial_translator.py"
    click Logger "https://github.com/ankitkr8540/pylinguist-paper/blob/master/pylinguist/utils/logger.py"
    click ResultsStore "https://github.com/ankitkr8540/pylinguist-paper/tree/master/Results/"
    click Notebook "https://github.com/ankitkr8540/pylinguist-paper/blob/master/Visualization.ipynb"
    click VizScripts "https://github.com/ankitkr8540/pylinguist-paper/tree/master/visualizations/"

    %% Styles
    classDef core fill:#BBDEFB,stroke:#1E88E5,color:#0D47A1
    classDef dataStore fill:#C8E6C9,stroke:#43A047,color:#1B5E20
    classDef external fill:#FFE0B2,stroke:#FB8C00,color:#E65100
    classDef ui fill:#E1BEE7,stroke:#8E24AA,color:#4A148C
```
