# gordicaleksa/open-nllb architecture

> Architecture diagram of the GitHub repository gordicaleksa/open-nllb, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-13). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/gordicaleksa/open-nllb
- Repository: https://github.com/gordicaleksa/open-nllb

## Overview

##### Project Type and Purpose
- This is an open-source ML framework and toolkit (built on Facebook’s fairseq) for training, fine-tuning, evaluating, and deploying large multilingual translation models (NLLB). It is not a traditional web app, but a highly modular research codebase comprising:
- Core library (`fairseq/`): data loading, model definitions, training loops, distributed support, optimizers, criterions, CLI interfaces.
- Examples (`examples/`): dozens of reproducible pipelines (NLLB training, LASER3 distillation, backtranslation, etc.).
- Scripts (`scripts/`): standalone utilities (checkpoint averaging, data conversion, BPE training).
- CLI (`fairseq_cli/`, top-level `train.py`, `generate.py`): user entrypoints.
- CI/CD (`.github/`, `.circleci/`): build, lint, test workflows.

##### File-Structure Analysis & Architectural Patterns
- Top-level directories separate concerns:
- `fairseq/`: the core engine (Modules, Models, Data, Tasks, Optim, Distributed).
- `fairseq_cli/`: command-line interface wrappers around core.
- `examples/`: end-to-end pipelines showing how to assemble config + core for specific tasks.
- `scripts/`: one-off and utility scripts.
- `tests/`: unit and integration tests.
- Configuration driven by Hydra and YAML files everywhere (`config/`, `examples/*/*.yaml`), supporting plugin patterns and registry-based discovery (`@register_task`, `@register_criterion`, etc.).
- Distributed training support via PyTorch DDP/FSDP and TPU wrappers in `fairseq/distributed`.
- Modular plugin approach: data sources, tokenizers, criterions, models, schedulers, and optimizers are all pluggable.

##### Key Components & Interactions (to include
A. Core Engine (`fairseq/`)
1. Data Layer (`fairseq/data/`):
- Data loading & preprocessing (tokenizers, SPM, indexed datasets).
- Dataset wrappers (monolingual, bitext, audio, multimodal).
2. Model Layer (`fairseq/models/`, `fairseq/modules/`):
- Transformers, MoE, wav2vec, speech modules, TTS, etc.
3. Task & Criterion (`fairseq/tasks/`, `fairseq/criterions/`):
- Training/evaluation loops, loss functions.
4. Optimization (`fairseq/optim/`, `fairseq/optim/lr_scheduler/`):
- Adam, Adafactor, LR schedules.
5. Distributed & Parallelism (`fairseq/distributed/`, `fairseq/model_parallel/`):
- Multi-GPU, FSDP, TPU, pipeline parallel.
6. CLI & Config (`fairseq_cli/`, Hydra integration in `fairseq/config/`):
- Entrypoints (`train.py`, `generate.py`, `interactive.py`, `hydra_train.py`).
B. Examples & Pipelines (`examples/nllb/`, others)
- YAML configurations → invocations of CLI scripts → leverage core engine.
- Data acquisition (public bitext, mined, backtranslation) through `examples/nllb/data/*`.
- Distillation and evaluation workflows (`examples/nllb/modeling/`, `examples/nllb/evaluation/`).
C. Utility Scripts (`scripts/`)
- Checkpoint management, BLEU scoring, BPE training.
D. CI/CD (`.github/`, `.circleci/`)
- Workflows that lint, test across GPUs/CPUs, build wheels.

##### Architectural Patterns & Principles
- Plugin/Registry for extensibility (register tasks, criterions, models).
- Configuration-as-code (Hydra + YAML).
- Separation of concerns: data, model, task, optim, distributed.
- Distributed-first design: built-in DDP/FSDP/TPU support.
- Test harness covering unit, integration, GPU/CPU, distributed.

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% CLI Layer
    subgraph "CLI & Configuration"
        direction TB
        ConfigManager["Hydra Config & CLI\n(train.py, generate.py,\ninteractive.py, hydra_train.py)"]:::cli
    end

    %% Data Sources
    subgraph "Data Sources" 
        direction TB
        RawData["Raw Data\n(public bitext,\nmined, backtranslation)"]:::external
    end

    %% Core Engine
    subgraph "Core Engine"
        direction TB
        DataLayer["Data Loader\n& Preprocessing"]:::core
        ModelLayer["Model Layer\n(Transformers, MoE,\nwav2vec, TTS)"]:::core
        TaskCriterion["Task & Criterion\n(Training/Eval loops,\nloss funcs)"]:::core
        Optimization["Optimizer & LR Scheduler\n(Adam, Adafactor,\nschedules)"]:::core
        DistributedEngine["Distributed & Parallelism\n(DDP, FSDP, TPU,\npipeline)"]:::core
        Registry["Plugin Registry\n(@register_task,\n@register_model)"]:::core
    end

    %% Example Pipelines
    subgraph "Example Pipelines"
        direction TB
        NLLBExamples["NLLB Examples\n(configs, distillation,\nevaluation)"]:::examples
    end

    %% Utility Scripts
    UtilityScripts["Utility Scripts\n(checkpoint avg,\nBLEU scoring, BPE)"]:::examples

    %% CI/CD & Storage
    subgraph "CI/CD & Artifact Store"
        direction TB
        CICD["CI/CD Workflows\n(.github, .circleci)"]:::cicd
        CheckpointSink["Checkpoint Sink\n(disk, S3, HF Hub)"]:::cicd
    end

    %% External Integrations
    subgraph "External Systems"
        direction TB
        Stopes["Stopes (data filtering)"]:::external
        LASER3["LASER3 (teacher encoder)"]:::external
        HFHub["Hugging Face Hub"]:::external
    end

    %% Flows
    ConfigManager -->|loads YAML| DataLayer
    DataLayer -->|batches| ModelLayer
    ModelLayer -->|forward/backward| TaskCriterion
    TaskCriterion -->|compute gradients| Optimization
    Optimization -->|update weights| DistributedEngine
    DistributedEngine -->|sync gradients| CheckpointSink
    CheckpointSink -->|upload| HFHub
    RawData -->|raw inputs| DataLayer
    NLLBExamples -->|invoke CLI| ConfigManager
    NLLBExamples -->|use data| RawData
    UtilityScripts -->|manage| CheckpointSink
    ConfigManager -->|register| Registry
    Registry --> DataLayer
    Registry --> ModelLayer
    Registry --> TaskCriterion
    Registry --> Optimization

    %% External hooks
    DataLayer -->|filter| Stopes
    ModelLayer -->|teacher outputs| LASER3

    %% Click Events
    click DataLayer "https://github.com/gordicaleksa/open-nllb/tree/nllb_replication/fairseq/data/"
    click ModelLayer "https://github.com/gordicaleksa/open-nllb/tree/nllb_replication/fairseq/models/"
    click Registry "https://github.com/gordicaleksa/open-nllb/tree/nllb_replication/fairseq/tasks/"
    click TaskCriterion "https://github.com/gordicaleksa/open-nllb/tree/nllb_replication/fairseq/criterions/"
    click Optimization "https://github.com/gordicaleksa/open-nllb/tree/nllb_replication/fairseq/optim/"
    click DistributedEngine "https://github.com/gordicaleksa/open-nllb/tree/nllb_replication/fairseq/distributed/"
    click ConfigManager "https://github.com/gordicaleksa/open-nllb/tree/nllb_replication/fairseq_cli/"
    click ConfigManager "https://github.com/gordicaleksa/open-nllb/tree/nllb_replication/fairseq/config/"
    click NLLBExamples "https://github.com/gordicaleksa/open-nllb/tree/nllb_replication/examples/nllb/"
    click UtilityScripts "https://github.com/gordicaleksa/open-nllb/tree/nllb_replication/scripts/"
    click CICD "https://github.com/gordicaleksa/open-nllb/tree/nllb_replication/ .github/"
    click CICD "https://github.com/gordicaleksa/open-nllb/tree/nllb_replication/.circleci/"

    %% Styles
    classDef core fill:#add8e6,stroke:#333,stroke-width:1px
    classDef examples fill:#90ee90,stroke:#333,stroke-width:1px
    classDef cli fill:#ffa500,stroke:#333,stroke-width:1px
    classDef cicd fill:#dda0dd,stroke:#333,stroke-width:1px
    classDef external fill:#d3d3d3,stroke:#333,stroke-width:1px
```
