# youssefkamel771/multilingual_machine_translation architecture

> Architecture diagram of the GitHub repository youssefkamel771/multilingual_machine_translation, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-08). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/youssefkamel771/multilingual_machine_translation
- Repository: https://github.com/youssefkamel771/multilingual_machine_translation

## Overview

##### Project Type and Purpose
- This is a **full-stack machine translation application** with two main deployment channels:
- A FastAPI-based backend serving custom encoder-decoder and transformer models.
- A Streamlit-based frontend using the pretrained `facebook/m2m100_418M` model for language-detection + translation.
- Purpose: Train multiple translation models (Italian, Arabic, Japanese, French, Portuguese) and deploy them for REST and web UI consumption.

##### Main Architectural Components
a. Data Layer
- Raw text corpora under `data/all` (e.g., `ita.txt`, `por.txt`, plus nonbreaking_prefixes).
- Preprocessing utilities in `utils/` (e.g., `utils_ita.py`, `utils_por.py`, `utils_fr.py`).
b. Training Layer
- Notebook experiments in `Notebooks/` for attention and transformer prototypes.
- Training scripts (`models/model_ita.py`, `models/model_por.py`) invoked by `main.py` (or separate training driver).
- Outputs stored under `artifacts/` (checkpoints, tokenizers, encoders/decoders, language pickles).
c. Inference Layer (API)
- `main.py` exposes FastAPI endpoints: `/translate` for custom models.
- Internally uses the trained artifacts, loads encoders/decoders and tokenizers.
d. Inference Layer (Web UI)
- `translation_app.py` (Streamlit) that:
- Accepts free‐form text.
- Calls Hugging Face’s `facebook/m2m100_418M` for language detection + translation.
e. External Services & Libraries
- Hugging Face Transformers library for pretrained model.
- FastAPI + Uvicorn for REST.
- Streamlit for UI.
- PyTorch (training & inference).

##### Relationships & Data Flows
- Data Flow A (Training):
data/all → utils preprocess → model training scripts → artifacts.
- Data Flow B (FastAPI Inference):
user request → FastAPI (`main.py`) → load relevant artifact → run translation → return response.
- Data Flow C (Streamlit UI):
user input → Streamlit app (`translation_app.py`) → pretrained model call → display detected language + translation.

##### Architectural Patterns & Principles
- **Separation of Concerns**:
- Data preprocessing (utils/)
- Model definition & training (models/, Notebooks/)
- Model storage (artifacts/)
- API serving vs. UI serving
- **Micro-service Style**:
- One service for custom-model translation (FastAPI)
- Another for general multilingual translation (Streamlit + pretrained)
- **Extensibility**:
- New language modules under `models/` + `utils/`
- New endpoints in `main.py`

##### Essential Elements
- Clear, concise component labels.
- Code modules vs. data stores vs. external services vs. user interface.
- Notes on ports/endpoints (e.g., FastAPI at `:8000/translate`, Streamlit on `:8501`).

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data Layer
    subgraph "Data Layer"
        DS(("Raw Text Corpora")):::data
        subgraph "Preprocessing Modules"
            U1["utils_ita.py"]:::code
            U2["utils_por.py"]:::code
            U3["utils_fr.py"]:::code
        end
    end

    %% Training Layer
    subgraph "Training Layer"
        NB["Notebooks"]:::code
        TM1["model_ita.py"]:::code
        TM2["model_por.py"]:::code
    end

    %% Artifact Store
    subgraph "Artifact Store"
        AM1(("Italian Model Artifacts")):::artifact
        AM2(("Portuguese Model Artifacts")):::artifact
    end

    %% Inference Layer
    subgraph "Inference"
        subgraph "FastAPI Service"
            FA["FastAPI Service\n:8000/translate"]:::service
        end
        subgraph "Streamlit Service"
            ST["Streamlit UI\n:8501"]:::ui
        end
    end

    %% External Service
    EXTERNAL["Hugging Face\nfacebook/m2m100_418M"]:::external

    %% Clients
    UserClient(["User\n(REST Client)"]):::ui
    Browser(["User\n(Web Browser)"]):::ui

    %% Data Flows: Training
    DS -->|"raw text"| U1
    DS -->|"raw text"| U2
    DS -->|"raw text"| U3
    U1 -->|"preprocessed data"| NB
    U2 -->|"preprocessed data"| NB
    U3 -->|"preprocessed data"| NB
    NB -->|"experiment insights"| TM1
    NB -->|"experiment insights"| TM2
    TM1 -->|"checkpoint .pt, tokenizer .pkl"| AM1
    TM2 -->|"checkpoint .pt, tokenizer .pkl"| AM2

    %% Data Flows: FastAPI Inference
    UserClient -->|"text request"| FA
    FA -->|"load model"| AM1
    FA -->|"translation response (JSON)"| UserClient

    %% Data Flows: Streamlit UI
    Browser -->|"input text"| ST
    ST -->|"transformers call"| EXTERNAL
    EXTERNAL -->|"translation + lang detection"| ST

    %% Click Events
    click DS "https://github.com/youssefkamel771/multilingual_machine_translation/tree/main/data/all/"
    click U1 "https://github.com/youssefkamel771/multilingual_machine_translation/blob/main/utils/utils_ita.py"
    click U2 "https://github.com/youssefkamel771/multilingual_machine_translation/blob/main/utils/utils_por.py"
    click U3 "https://github.com/youssefkamel771/multilingual_machine_translation/blob/main/utils/utils_fr.py"
    click NB "https://github.com/youssefkamel771/multilingual_machine_translation/tree/main/Notebooks/"
    click TM1 "https://github.com/youssefkamel771/multilingual_machine_translation/blob/main/models/model_ita.py"
    click TM2 "https://github.com/youssefkamel771/multilingual_machine_translation/blob/main/models/model_por.py"
    click AM1 "https://github.com/youssefkamel771/multilingual_machine_translation/tree/main/artifacts/ita_model/"
    click AM2 "https://github.com/youssefkamel771/multilingual_machine_translation/tree/main/artifacts/por_model/"
    click FA "https://github.com/youssefkamel771/multilingual_machine_translation/blob/main/main.py"
    click ST "https://github.com/youssefkamel771/multilingual_machine_translation/blob/main/translation_app.py"

    %% Styles
    classDef data fill:#ccffcc,stroke:#006600;
    classDef artifact fill:#ccffcc,stroke:#006600,shape:cylinder;
    classDef code fill:#cce5ff,stroke:#004085;
    classDef service fill:#e2e3e5,stroke:#6c757d;
    classDef ui fill:#f8d7da,stroke:#721c24;
    classDef external fill:#fff3cd,stroke:#856404;
```
