How gordicaleksa/open-nllb works
Project Type and Purpose
• This is an open-source ML framework and toolkit (built on Facebook’s fairseq) for training, fine-tuning, evaluating, and deploying large multilingual translation models (NLLB). It is not a traditional web app, but a highly modular research codebase comprising:
• Core library (fairseq/): data loading, model definitions, training loops, distributed support, optimizers, criterions, CLI interfaces.
• Examples (examples/): dozens of reproducible pipelines (NLLB training, LASER3 distillation, backtranslation, etc.).
• Scripts (scripts/): standalone utilities (checkpoint averaging, data conversion, BPE training).
• CLI (fairseq_cli/, top-level train.py, generate.py): user entrypoints.
• CI/CD (.github/, .circleci/): build, lint, test workflows.
File-Structure Analysis & Architectural Patterns
• Top-level directories separate concerns:
• fairseq/: the core engine (Modules, Models, Data, Tasks, Optim, Distributed).
• fairseq_cli/: command-line interface wrappers around core.
• examples/: end-to-end pipelines showing how to assemble config + core for specific tasks.
• scripts/: one-off and utility scripts.
• tests/: unit and integration tests.
• Configuration driven by Hydra and YAML files everywhere (config/, examples/*/*.yaml), supporting plugin patterns and registry-based discovery (@register_task, @register_criterion, etc.).
• Distributed training support via PyTorch DDP/FSDP and TPU wrappers in fairseq/distributed.
• Modular plugin approach: data sources, tokenizers, criterions, models, schedulers, and optimizers are all pluggable.
Key Components & Interactions (to include
A. Core Engine (fairseq/)
1. Data Layer (fairseq/data/):
• Data loading & preprocessing (tokenizers, SPM, indexed datasets).
• Dataset wrappers (monolingual, bitext, audio, multimodal).
2. Model Layer (fairseq/models/, fairseq/modules/):
• Transformers, MoE, wav2vec, speech modules, TTS, etc.
3. Task & Criterion (fairseq/tasks/, fairseq/criterions/):
• Training/evaluation loops, loss functions.
4. Optimization (fairseq/optim/, fairseq/optim/lr_scheduler/):
• Adam, Adafactor, LR schedules.
5. Distributed & Parallelism (fairseq/distributed/, fairseq/model_parallel/):
• Multi-GPU, FSDP, TPU, pipeline parallel.
6. CLI & Config (fairseq_cli/, Hydra integration in fairseq/config/):
• Entrypoints (train.py, generate.py, interactive.py, hydra_train.py).
B. Examples & Pipelines (examples/nllb/, others)
• YAML configurations → invocations of CLI scripts → leverage core engine.
• Data acquisition (public bitext, mined, backtranslation) through examples/nllb/data/*.
• Distillation and evaluation workflows (examples/nllb/modeling/, examples/nllb/evaluation/).
C. Utility Scripts (scripts/)
• Checkpoint management, BLEU scoring, BPE training.
D. CI/CD (.github/, .circleci/)
• Workflows that lint, test across GPUs/CPUs, build wheels.
Architectural Patterns & Principles
• Plugin/Registry for extensibility (register tasks, criterions, models).
• Configuration-as-code (Hydra + YAML).
• Separation of concerns: data, model, task, optim, distributed.
• Distributed-first design: built-in DDP/FSDP/TPU support.
• Test harness covering unit, integration, GPU/CPU, distributed.