# facebookresearch/lingua architecture

> Architecture diagram of the GitHub repository facebookresearch/lingua, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/facebookresearch/lingua
- Repository: https://github.com/facebookresearch/lingua

## Overview

##### Project Type and Purpose
- Meta Lingua is a minimal and fast LLM (Large Language Model) training and inference library built on PyTorch.
- Its purpose is to allow researchers to experiment with new architectures, losses, and data configurations. It provides “apps” that serve as templates to run training, evaluation, generation, and debugging jobs on LLMs.
- The README indicates multiple experimental “apps” (e.g., main, fastRNN, mamba, mtp, and plots), each orchestrating different model architectures and experiments.

##### File Structure
- The top-level directories are:
- “lingua”: The core library containing reusable modules such as data loading (data.py), distributed training (distributed.py), checkpoint handling (checkpoint.py), logging, tokenization, and profiling.
- “apps”: A collection of applications that combine these core modules:
∘ “main” – the main language modeling app based on a transformer architecture.
∘ “fastRNN” – contains variations like hawk, minGRU, and minLSTM components.
∘ “mamba” – another app with components like causal_conv1d and ssm.
∘ “mtp” – focused on multi-token prediction.
∘ “plots” – scripts for analysis and visualization of training metrics.
- “setup”: Contains scripts for environment creation and data/tokenizer preparation.
- Configuration files (e.g., YAMLs in various configs directories) and setup scripts indicate that the system is highly configurable and designed for distributed (SLURM) as well as local execution.

##### README for Additional Insights
- The README gives an overview of how different components cooperate during tasks like training, evaluation, and debugging.
- Important components like “transformer.py”, “distributed.py”, “data.py”, “profiling.py”, and “checkpoint.py” are explicitly described.
- It also explains how configuration management is done using OmegaConf and dataclasses, emphasizing the importance of adaptable and modular configurations.
- The dump directory structure described at the end shows how checkpoints, logs, and code backups are organized, indicating integration of runtime operations, recovery, and profiling.

a. Main Components:
- Core Library (lingua):
– Data Management: data.py, tokenizer.py.
– Model Components: transformer.py (model architecture), args.py (configuration management).
– Distributed Training: distributed.py (parallel and sharding utilities), stool.py (SLURM job management).
– Utilities: checkpoint.py (saving and loading), logging and metrics (logger.py, metrics.py), profiling.py (performance tracing).
- Applications (apps):
– Main App: apps/main (training & evaluation scripts and configurations for transformer-based LLM).
– Experimental Architectures: apps/fastRNN (with components for hawk, minGRU, minLSTM), apps/mamba (for alternative architectures like causal_conv and ssm), and apps/mtp (for multi-token prediction).
– Visualization: apps/plots (analysis and probe animation).
- Setup Environment:
– Setup scripts for creating environments and preparing data/tokenizers.
- External Dependencies:
– PyTorch, OmegaConf, and xformers profiling tool.
– SLURM for distributed job management.

b. Relationships and Interactions:
- The “apps” scripts act as the orchestration layer that combines components from “lingua” (common modules) with configurations (YAML files under respective apps).
- The training loop (e.g. in apps/main/train.py) makes use of:
– Data loader from lingua/data.py.
– Model defined in lingua/transformer.py.
– Distributed utilities from lingua/distributed.py.
– Checkpoint and logging utilities.
- The experimental “fastRNN”, “mamba”, and “mtp” apps leverage custom components (located under their respective directories) that plug into the core functionalities.
- SLURM-related scripts and “stool” wrap commands, manage distributed job launches, and backup code snapshots.

c. Architectural Patterns/Design Principles:
- Modularity: Separation between core library (lingua) and experimental apps.
- Configurability: Extensive use of YAML configuration files and OmegaConf for command-line overrides.
- Distributed Training: Abstractions in distributed.py and stool.py demonstrate design for parallel and flexible execution.
- Reusability: Reusable components (data, transformer, checkpoint) are isolated to support rapid experimentation.

d. Relevant Technologies and Frameworks:
- PyTorch (for neural network modeling, training, and distributed computing).
- OmegaConf (for configuration management).
- SLURM (for job scheduling in distributed environments).
- xformers profiler integration for performance analysis.
- Scripting in Python tying all components.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Setup and External Dependencies
    subgraph "Setup & External"
        Setup["Setup: Environment Preparation"]:::setup
        External["External Dependencies: PyTorch,OmegaConf,SLURM,xformers"]:::external
        Setup -->|"configures"| External
    end

    %% Applications Layer
    subgraph "Applications"
        MainApp["Main App (Transformer-based LLM)"]:::app
        FastRNN["Experimental fastRNN"]:::app
        Mamba["Experimental Mamba"]:::app
        MTP["Experimental MTP"]:::app
        Plots["Visualization & Analysis"]:::app
        %% Setup triggers the apps
        Setup -->|"initiates"| MainApp
        Setup -->|"initiates"| FastRNN
        Setup -->|"initiates"| Mamba
        Setup -->|"initiates"| MTP
        Setup -->|"initiates"| Plots
    end

    %% Core Library Layer
    subgraph "Core Library"
        %% Data Management Subgroup
        subgraph "Data Management"
            Data["data.py"]:::core
            Tokenizer["tokenizer.py"]:::core
        end
        %% Model Components & Configuration Subgroup
        subgraph "Model Components & Config"
            Transformer["transformer.py"]:::core
            Args["args.py"]:::core
        end
        %% Distributed Training Utilities Subgroup
        subgraph "Distributed Training"
            Distributed["distributed.py"]:::core
            Stool["stool.py"]:::core
        end
        %% Checkpointing & Logging Utilities Subgroup
        subgraph "Checkpointing & Logging"
            Checkpoint["checkpoint.py"]:::core
            Logger["logger.py"]:::core
            Metrics["metrics.py"]:::core
            Profiling["profiling.py"]:::core
        end
    end

    %% Interactions from Applications to Core Library Components
    MainApp -->|"calls"| Data
    MainApp -->|"calls"| Transformer
    MainApp -->|"calls"| Distributed
    MainApp -->|"calls"| Checkpoint

    FastRNN -->|"extends"| Transformer
    FastRNN -->|"calls"| Data

    Mamba -->|"uses"| Transformer
    Mamba -->|"calls"| Distributed

    MTP -->|"uses"| Transformer
    MTP -->|"calls"| Data

    %% Visualization interacts with Checkpointing & Logging
    Plots -->|"analyzes"| Logger
    Plots -->|"analyzes"| Metrics
    Plots -->|"analyzes"| Profiling

    %% External Dependencies support Core Library Components
    External -->|"supports"| Transformer
    External -->|"supports"| Distributed
    Stool -->|"interfaces"| External

    %% Click Events
    click Data "https://github.com/facebookresearch/lingua/blob/main/lingua/data.py"
    click Tokenizer "https://github.com/facebookresearch/lingua/blob/main/lingua/tokenizer.py"
    click Transformer "https://github.com/facebookresearch/lingua/blob/main/lingua/transformer.py"
    click Args "https://github.com/facebookresearch/lingua/blob/main/lingua/args.py"
    click Distributed "https://github.com/facebookresearch/lingua/blob/main/lingua/distributed.py"
    click Stool "https://github.com/facebookresearch/lingua/blob/main/lingua/stool.py"
    click Checkpoint "https://github.com/facebookresearch/lingua/blob/main/lingua/checkpoint.py"
    click Logger "https://github.com/facebookresearch/lingua/blob/main/lingua/logger.py"
    click Metrics "https://github.com/facebookresearch/lingua/blob/main/lingua/metrics.py"
    click Profiling "https://github.com/facebookresearch/lingua/blob/main/lingua/profiling.py"
    click MainApp "https://github.com/facebookresearch/lingua/tree/main/apps/main"
    click FastRNN "https://github.com/facebookresearch/lingua/tree/main/apps/fastRNN"
    click Mamba "https://github.com/facebookresearch/lingua/tree/main/apps/mamba"
    click MTP "https://github.com/facebookresearch/lingua/tree/main/apps/mtp"
    click Plots "https://github.com/facebookresearch/lingua/tree/main/apps/plots"
    click Setup "https://github.com/facebookresearch/lingua/tree/main/setup"

    %% Styles
    classDef core fill:#AED6F1,stroke:#1F618D,stroke-width:2px;
    classDef app fill:#F9E79F,stroke:#B9770E,stroke-width:2px;
    classDef setup fill:#D5F5E3,stroke:#1E8449,stroke-width:2px;
    classDef external fill:#FADBD8,stroke:#C0392B,stroke-width:2px;
```
