How youssefkamel771/multilingual_machine_translation works
Project Type and Purpose
• This is a full-stack machine translation application with two main deployment channels:
• A FastAPI-based backend serving custom encoder-decoder and transformer models.
• A Streamlit-based frontend using the pretrained facebook/m2m100_418M model for language-detection + translation.
• Purpose: Train multiple translation models (Italian, Arabic, Japanese, French, Portuguese) and deploy them for REST and web UI consumption.
Main Architectural Components
a. Data Layer
• Raw text corpora under data/all (e.g., ita.txt, por.txt, plus nonbreaking_prefixes).
• Preprocessing utilities in utils/ (e.g., utils_ita.py, utils_por.py, utils_fr.py).
b. Training Layer
• Notebook experiments in Notebooks/ for attention and transformer prototypes.
• Training scripts (models/model_ita.py, models/model_por.py) invoked by main.py (or separate training driver).
• Outputs stored under artifacts/ (checkpoints, tokenizers, encoders/decoders, language pickles).
c. Inference Layer (API)
• main.py exposes FastAPI endpoints: /translate for custom models.
• Internally uses the trained artifacts, loads encoders/decoders and tokenizers.
d. Inference Layer (Web UI)
• translation_app.py (Streamlit) that:
• Accepts free‐form text.
• Calls Hugging Face’s facebook/m2m100_418M for language detection + translation.
e. External Services & Libraries
• Hugging Face Transformers library for pretrained model.
• FastAPI + Uvicorn for REST.
• Streamlit for UI.
• PyTorch (training & inference).
Relationships & Data Flows
• Data Flow A (Training):
data/all → utils preprocess → model training scripts → artifacts.
• Data Flow B (FastAPI Inference):
user request → FastAPI (main.py) → load relevant artifact → run translation → return response.
• Data Flow C (Streamlit UI):
user input → Streamlit app (translation_app.py) → pretrained model call → display detected language + translation.
Architectural Patterns & Principles
• Separation of Concerns:
• Data preprocessing (utils/)
• Model definition & training (models/, Notebooks/)
• Model storage (artifacts/)
• API serving vs. UI serving
• Micro-service Style:
• One service for custom-model translation (FastAPI)
• Another for general multilingual translation (Streamlit + pretrained)
• Extensibility:
• New language modules under models/ + utils/
• New endpoints in main.py
Essential Elements
• Clear, concise component labels.
• Code modules vs. data stores vs. external services vs. user interface.
• Notes on ports/endpoints (e.g., FastAPI at :8000/translate, Streamlit on :8501).