# yerevann/ynntitan architecture

> Architecture diagram of the GitHub repository yerevann/ynntitan, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-01-07). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/yerevann/ynntitan
- Repository: https://github.com/yerevann/ynntitan

## Overview

##### Main Components

a) Training Pipeline Layer:
- Training Orchestration (train.py as entry point)
- Configuration Management (config_manager.py)
- Checkpoint Management (checkpoint.py)
- Metrics & Logging (metrics.py, logging.py)

b) Model Layer:
- Model Implementations
- Llama Models
- OPT Models
- Normalization Components (norms.py)
- Tokenizer System

c) Parallelization Layer:
- FSDP (Fully Sharded Data Parallel)
- Tensor Parallel
- Pipeline Parallel
- Data Parallel

d) Optimization Layer:
- Float8 Support
- Optimizer Implementation
- Profiling System

##### Component Relationships

**┌─────────────────────────────────────────────────────────┐**
- Training Pipeline                       │
- ┌──────────┐  ┌──────────┐  ┌──────────┐  ┌──────────┐│
- Config  │  │Checkpoint│  │ Metrics  │  │ Logging  ││
- ┘  └──────────┘  └──────────┘  └──────────┘│
- ┬─────────────────────────────────┘
- ↓
**┌─────────────────────────────────────────────────────────┐**
- Parallelization Layer                   │
- ┌──────────┐  ┌──────────┐  ┌──────────┐  ┌──────────┐│
- FSDP   │  │  Tensor  │  │ Pipeline │  │   Data   ││
- Parallel │  │ Parallel │  │ Parallel ││
- ┘  └──────────┘  └──────────┘  └──────────┘│
- ┬─────────────────────────────────┘
- ↓
**┌─────────────────────────────────────────────────────────┐**
- Model Layer                          │
- ┌──────────┐  ┌──────────┐  ┌──────────┐  ┌──────────┐│
- Llama   │  │   OPT    │  │Tokenizers│  │  Norms   ││
- Models  │  │  Models  │  │          │  │          ││
- ┘  └──────────┘  └──────────┘  └──────────┘│
- ┬─────────────────────────────────┘
- ↓
**┌─────────────────────────────────────────────────────────┐**
- Optimization Layer                      │
- ┌──────────┐  ┌──────────┐  ┌──────────┐              │
- Float8  │  │Optimizer │  │Profiling │              │

- ┘  └──────────┘  └──────────┘              │
- ┘

##### Important Details

a) Data Flow Indicators:
- How data flows from configuration through the parallelization layer
- Checkpoint save/load paths
- Model output flow to metrics collection

b) Integration Points:
- PyTorch integration points
- External dependencies (e.g., tokenizer models)
- Monitoring/logging interfaces (Aim integration)

##### Implementation Notes

- Callouts for upcoming features (async checkpointing, context parallel, etc.)
- Notes about interoperability with torchtune

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Styles
    classDef training fill:#3498db,stroke:#2980b9,color:white
    classDef parallel fill:#2ecc71,stroke:#27ae60,color:white
    classDef model fill:#e67e22,stroke:#d35400,color:white
    classDef optimization fill:#9b59b6,stroke:#8e44ad,color:white
    
    %% Training Pipeline Layer
    subgraph Training
        Train["Training Orchestration"]
        Config["Configuration Management"]
        Checkpoint["Checkpoint Management"]
        Metrics["Metrics"]
        Logging["Logging"]
    end
    
    %% Parallelization Layer
    subgraph Parallel
        FSDP["FSDP"]
        TensorParallel["Tensor Parallel"]
        PipelineParallel["Pipeline Parallel"]
        DataParallel["Data Parallel"]
        ParallelUtils["Parallel Utils"]
    end
    
    %% Model Layer
    subgraph Models
        Llama["Llama Models"]
        OPT["OPT Models"]
        Tokenizers["Tokenizer System"]
        Norms["Normalization"]
    end
    
    %% Optimization Layer
    subgraph Optimization
        Float8["Float8 Support"]
        Optimizer["Optimizer"]
        Profiling["Profiling System"]
    end
    
    %% Relationships
    Training --> Parallel
    Parallel --> Models
    Models --> Optimization
    
    %% Click Events
    click Train "https://github.com/YerevaNN/YNNtitan/blob/main/train.py"
    click Config "https://github.com/YerevaNN/YNNtitan/blob/main/torchtitan/config_manager.py"
    click Checkpoint "https://github.com/YerevaNN/YNNtitan/blob/main/torchtitan/checkpoint.py"
    click Metrics "https://github.com/YerevaNN/YNNtitan/blob/main/torchtitan/metrics.py"
    click Logging "https://github.com/YerevaNN/YNNtitan/blob/main/torchtitan/logging.py"
    click Llama "https://github.com/YerevaNN/YNNtitan/blob/main/torchtitan/models/llama/model.py"
    click OPT "https://github.com/YerevaNN/YNNtitan/blob/main/torchtitan/models/opt/model.py"
    click Norms "https://github.com/YerevaNN/YNNtitan/blob/main/torchtitan/models/norms.py"
    click Tokenizers "https://github.com/YerevaNN/YNNtitan/tree/main/torchtitan/tokenizers/tokenizer"
    click PipelineParallel "https://github.com/YerevaNN/YNNtitan/blob/main/torchtitan/parallelisms/pipeline_llama.py"
    click TensorParallel "https://github.com/YerevaNN/YNNtitan/blob/main/torchtitan/parallelisms/parallel_dims.py"
    click FSDP "https://github.com/YerevaNN/YNNtitan/blob/main/torchtitan/parallelisms/parallelize_llama.py"
    click ParallelUtils "https://github.com/YerevaNN/YNNtitan/blob/main/torchtitan/parallelisms/utils.py"
    click Float8 "https://github.com/YerevaNN/YNNtitan/blob/main/torchtitan/float8.py"
    click Optimizer "https://github.com/YerevaNN/YNNtitan/blob/main/torchtitan/optimizer.py"
    click Profiling "https://github.com/YerevaNN/YNNtitan/blob/main/torchtitan/profiling.py"
    
    %% Apply styles
    class Train,Config,Checkpoint,Metrics,Logging training
    class FSDP,TensorParallel,PipelineParallel,DataParallel,ParallelUtils parallel
    class Llama,OPT,Tokenizers,Norms model
    class Float8,Optimizer,Profiling optimization
```
