How stif83/translator works
Project Type and Purpose
• This is an open-source machine-learning translation tool (French ⇄ English) built in Python using NLTK.
• It provides two primary workflows: training a translation model and running inference (testing) against parallel text data.
Key Directories and Files
• data/raw/en-fr small.tmx: parallel corpus in TMX format (raw bilingual training data)
• src/base_model.py: abstract or shared logic for model definition, training, saving, loading, and inference
• main_train.py (invoked from README): orchestrates data loading, preprocessing, model instantiation, training loop
• main_test.py (invoked from README): loads a trained model, runs translation on test inputs, evaluates or outputs translations
• requirements.txt: Python dependencies (NLTK, numpy, etc.)
• README.md: setup instructions
Architectural Components
a. Data Ingestion & Storage
• Raw corpus loader (reads TMX)
• Preprocessor (tokenization, sentence alignment)
b. Model Layer
• BaseModel class (in src/base_model.py)
• Model Trainer (main_train.py workflow)
• Model Serializer (save/load checkpoints)
c. Inference Layer
• Model Loader (loads serialized model)
• Translator / Inference Engine (main_test.py workflow)
d. Environment & Dependencies
• Python Virtual Environment (venv)
• requirements.txt (dependency management)
Component Interactions
• Training Data → Model Trainer → Trained Model
• Loaded Model → Inference Engine → Translated Output
Architectural Patterns & Principles
• Modular separation: data prep, model logic, training, and inference are distinct modules/files
• Single Responsibility: base_model.py defines core ML functions; main_*.py scripts orchestrate flows
• Reusability: BaseModel can be extended for alternative translation strategies
• Batch pipelines: clear flow of data from raw corpus through to final translation