# facebookresearch/large_concept_model architecture

> Architecture diagram of the GitHub repository facebookresearch/large_concept_model, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2024-12-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/facebookresearch/large_concept_model
- Repository: https://github.com/facebookresearch/large_concept_model

## Overview

##### Main Components

a) Core Architecture Components:
- SONAR Embedding Space (Input/Output Layer)
- Model Variants:
- Base MSE LCM
- Two-Tower Diffusion LCM
- (Future) Quantized SONAR Space Models
- Training Pipeline
- Inference Pipeline
- Evaluation System

b) Supporting Components:
- Data Processing Pipeline
- Model Registry
- Normalization System
- Metrics Collection

##### Key Relationships to Highlight

a) Data Flow:
**Raw Text → Sentence Splitting → SONAR Embedding → Normalized Embeddings → Model Training/Inference → Generated Embeddings → Text Generation**

b) Training Pipeline:
**Data Loading → Batch Processing → Model Training → Checkpointing → Evaluation**

##### Architectural Patterns

- Modular Architecture (evident from the /lcm directory structure)
- Pipeline Pattern (for data processing and evaluation)
- Factory Pattern (seen in nn/denoisers/factory.py and nn/transformer/factory.py)

##### Implementation Details

- LCM System                                           |

- ↓

- Data Pipeline |    |  Model Core    |    |   Inference   |    |    Evaluation     |

- Sentence Split|    |- Base LCM      |    |- Generator    |    |- Metrics         |
- SONAR Embed   |    |- Two-Tower LCM |    |- Scorer      |    |- Tasks           |
- Normalize     |    |- Neural Nets   |    |- Batching     |    |- Predictors     |

- ↓

- Infrastructure Layer                                       |

- Training Systems (Local/SLURM) | Logging | Monitoring                |

##### Important Details

- Model Architecture Variants (1.6B parameters)
- Training Data Scale (1.3T tokens)
- SONAR Integration (200 languages support)
- Distributed Training Support
- Evaluation Metrics Flow

## Diagram (Mermaid)

```mermaid
graph TB
    subgraph Legend
        L1[Process]
        L2[(Database)]
        L3{{External System}}
        L4[/API Gateway/]
    end

    subgraph "Data Processing Layer"
        RawText[Raw Text]
        SentenceSplit["Sentence Splitting"]:::data
        SonarEmbed["SONAR Embedding"]:::data
        Normalize["Normalization"]:::data
    end

    subgraph "Model Layer"
        BaseLCM["Base MSE LCM"]:::model
        TwoTowerLCM["Two-Tower Diffusion LCM"]:::model
        NeuralNets["Neural Networks"]:::model
    end

    subgraph "Training System"
        TrainPipe["Training Pipeline"]:::train
        ModelConfig["Model Configuration"]:::train
        TrainRecipes["Training Recipes"]:::train
    end

    subgraph "Inference Layer"
        InferenceSystem["Inference System"]:::inference
        Generator["Generator"]:::inference
        Scorer["Scorer"]:::inference
    end

    subgraph "Evaluation Layer"
        EvalSystem["Evaluation System"]:::eval
        Metrics["Metrics Collection"]:::eval
        Tasks["Evaluation Tasks"]:::eval
        Predictors["Model Predictors"]:::eval
        EvalRecipes["Evaluation Recipes"]:::eval
    end

    subgraph "Infrastructure"
        SLURM{{SLURM}}:::external
        SONAR{{SONAR}}:::external
        Monitoring["Monitoring & Logging"]:::infra
    end

    RawText --> SentenceSplit
    SentenceSplit --> SonarEmbed
    SonarEmbed --> Normalize
    Normalize --> |Training Data| TrainPipe
    
    TrainPipe --> BaseLCM
    TrainPipe --> TwoTowerLCM
    ModelConfig --> TrainPipe
    TrainRecipes --> TrainPipe
    
    BaseLCM --> InferenceSystem
    TwoTowerLCM --> InferenceSystem
    NeuralNets --> BaseLCM
    NeuralNets --> TwoTowerLCM
    
    InferenceSystem --> Generator
    InferenceSystem --> Scorer
    
    Generator --> EvalSystem
    Scorer --> EvalSystem
    
    EvalSystem --> Metrics
    EvalSystem --> Tasks
    EvalSystem --> Predictors
    EvalRecipes --> EvalSystem
    
    SLURM -.-> TrainPipe
    SONAR -.-> SonarEmbed
    
    Monitoring -.-> TrainPipe
    Monitoring -.-> InferenceSystem
    Monitoring -.-> EvalSystem

    %% Styles
    classDef model fill:#90EE90
    classDef data fill:#87CEEB
    classDef train fill:#FFB6C1
    classDef inference fill:#FFA500
    classDef eval fill:#DDA0DD
    classDef infra fill:#D3D3D3
    classDef external fill:#F0F0F0,stroke-dasharray: 5 5

    %% Click Events
    click BaseLCM "https://github.com/facebookresearch/large_concept_model/tree/main/lcm/models/base_lcm/"
    click TwoTowerLCM "https://github.com/facebookresearch/large_concept_model/tree/main/lcm/models/two_tower_diffusion_lcm/"
    click TrainPipe "https://github.com/facebookresearch/large_concept_model/tree/main/lcm/train/"
    click InferenceSystem "https://github.com/facebookresearch/large_concept_model/tree/main/lcm/inference/"
    click EvalSystem "https://github.com/facebookresearch/large_concept_model/tree/main/lcm/evaluation/"
    click SentenceSplit "https://github.com/facebookresearch/large_concept_model/blob/main/lcm/datasets/sentence_splitter_pipeline.py"
    click Normalize "https://github.com/facebookresearch/large_concept_model/tree/main/lcm/models/sonar_normalizer/"
    click Metrics "https://github.com/facebookresearch/large_concept_model/tree/main/lcm/evaluation/metrics/"
    click NeuralNets "https://github.com/facebookresearch/large_concept_model/tree/main/lcm/nn/"
    click Predictors "https://github.com/facebookresearch/large_concept_model/tree/main/lcm/evaluation/predictors/"
    click Tasks "https://github.com/facebookresearch/large_concept_model/tree/main/lcm/evaluation/tasks/"
    click ModelConfig "https://github.com/facebookresearch/large_concept_model/tree/main/lcm/cards/"
    click TrainRecipes "https://github.com/facebookresearch/large_concept_model/tree/main/recipes/train/"
    click EvalRecipes "https://github.com/facebookresearch/large_concept_model/blob/main/recipes/common/evals.yaml"
```
