How yerevann/ynntitan works
Main Components
a) Training Pipeline Layer:
• Training Orchestration (train.py as entry point)
• Configuration Management (config_manager.py)
• Checkpoint Management (checkpoint.py)
• Metrics & Logging (metrics.py, logging.py)
b) Model Layer:
• Model Implementations
• Llama Models
• OPT Models
• Normalization Components (norms.py)
• Tokenizer System
c) Parallelization Layer:
• FSDP (Fully Sharded Data Parallel)
• Tensor Parallel
• Pipeline Parallel
• Data Parallel
d) Optimization Layer:
• Float8 Support
• Optimizer Implementation
• Profiling System
Component Relationships
┌─────────────────────────────────────────────────────────┐
• Training Pipeline │
• ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐│
• Config │ │Checkpoint│ │ Metrics │ │ Logging ││
• ┘ └──────────┘ └──────────┘ └──────────┘│
• ┬─────────────────────────────────┘
• ↓
┌─────────────────────────────────────────────────────────┐
• Parallelization Layer │
• ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐│
• FSDP │ │ Tensor │ │ Pipeline │ │ Data ││
• Parallel │ │ Parallel │ │ Parallel ││
• ┘ └──────────┘ └──────────┘ └──────────┘│
• ┬─────────────────────────────────┘
• ↓
┌─────────────────────────────────────────────────────────┐
• Model Layer │
• ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐│
• Llama │ │ OPT │ │Tokenizers│ │ Norms ││
• Models │ │ Models │ │ │ │ ││
• ┘ └──────────┘ └──────────┘ └──────────┘│
• ┬─────────────────────────────────┘
• ↓
┌─────────────────────────────────────────────────────────┐
• Optimization Layer │
• ┌──────────┐ ┌──────────┐ ┌──────────┐ │
• Float8 │ │Optimizer │ │Profiling │ │
• ┘ └──────────┘ └──────────┘ │
• ┘
Important Details
a) Data Flow Indicators:
• How data flows from configuration through the parallelization layer
• Checkpoint save/load paths
• Model output flow to metrics collection
b) Integration Points:
• PyTorch integration points
• External dependencies (e.g., tokenizer models)
• Monitoring/logging interfaces (Aim integration)
Implementation Notes
• Callouts for upcoming features (async checkpointing, context parallel, etc.)
• Notes about interoperability with torchtune