How facebookresearch/large_concept_model works
Main Components
a) Core Architecture Components:
• SONAR Embedding Space (Input/Output Layer)
• Model Variants:
• Base MSE LCM
• Two-Tower Diffusion LCM
• (Future) Quantized SONAR Space Models
• Training Pipeline
• Inference Pipeline
• Evaluation System
b) Supporting Components:
• Data Processing Pipeline
• Model Registry
• Normalization System
• Metrics Collection
Key Relationships to Highlight
a) Data Flow:
Raw Text → Sentence Splitting → SONAR Embedding → Normalized Embeddings → Model Training/Inference → Generated Embeddings → Text Generation
b) Training Pipeline:
Data Loading → Batch Processing → Model Training → Checkpointing → Evaluation
Architectural Patterns
• Modular Architecture (evident from the /lcm directory structure)
• Pipeline Pattern (for data processing and evaluation)
• Factory Pattern (seen in nn/denoisers/factory.py and nn/transformer/factory.py)
Implementation Details
• LCM System |
• ↓
• Data Pipeline | | Model Core | | Inference | | Evaluation |
• Sentence Split| |- Base LCM | |- Generator | |- Metrics |
• SONAR Embed | |- Two-Tower LCM | |- Scorer | |- Tasks |
• Normalize | |- Neural Nets | |- Batching | |- Predictors |
• ↓
• Infrastructure Layer |
• Training Systems (Local/SLURM) | Logging | Monitoring |
Important Details
• Model Architecture Variants (1.6B parameters)
• Training Data Scale (1.3T tokens)
• SONAR Integration (200 languages support)
• Distributed Training Support
• Evaluation Metrics Flow