# work4ai/deepseek-from-scratch architecture

> Architecture diagram of the GitHub repository work4ai/deepseek-from-scratch, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-08-23). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/work4ai/deepseek-from-scratch
- Repository: https://github.com/work4ai/deepseek-from-scratch

## Overview

##### Project type and purpose
- This is an open-source machine-learning project that implements a transformer-based language model (DeepSeek V3) from scratch, complete with training and inference pipelines.
- Its purpose is to demonstrate and experiment with innovations (MLA, MoE, MTP) on top of a standard transformer, train on FineWeb-Edu, and serve text generation.

##### File structure
- Top-level scripts and config
– main.py: CLI entry-point for training or other modes
– run_inference.py: standalone inference runner
– prepare_data_fineweb.py / prepare_data_tiny_stories.py: data-prep scripts
– requirements.txt, .gitignore, README.md
- models/
– attention.py, moe.py, mtp.py: core algorithm modules
– layers.py: RoPE, RMSNorm, SwiGLU implementations
– model.py: assembles transformer blocks
– config.py: hyperparameters and architecture settings
- training/
– data_loader.py: dataset reading, tokenization, batching
– trainer.py: training loop, optimizer, learning rate schedulers, W&B logging
- inference/
– generator.py: sampling/speculative decoding logic
– run_inference.py: loads checkpoint, runs generator on prompts
- notebooks/: Jupyter demonstrations of each innovation

##### README for additional insights
- Highlights external dependencies: HuggingFace datasets, Weights & Biases for logging, PyTorch (implied).
- Shows model parameter counts, dataset sizes, training metrics.

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Data Preparation Layer
    subgraph "Data Preparation Layer"
        direction TB
        RawData1["Raw Data (FineWeb-Edu)"]:::storage
        RawData2["Raw Data (TinyStories)"]:::storage
        PrepFine["prepare_data_fineweb.py"]:::code
        PrepTiny["prepare_data_tiny_stories.py"]:::code
    end

    %% Model Definition Layer
    subgraph "Model Definition Layer"
        direction TB
        Config["models/config.py"]:::code
        ModelAssembly["models/model.py"]:::code
        MLA["Multi-Head Latent Attention Module\n(models/attention.py)"]:::code
        MoE["Mixture-of-Experts Module\n(models/moe.py)"]:::code
        MTP["Multi-Token Prediction Module\n(models/mtp.py)"]:::code
        Layers["Helper Layers\n(models/layers.py)"]:::code
    end

    %% Training Layer
    subgraph "Training Layer"
        direction TB
        DataLoader["training/data_loader.py"]:::code
        Trainer["training/trainer.py"]:::code
        Checkpoints["Model Checkpoints"]:::storage
        WB["Weights & Biases"]:::service
    end

    %% Inference Layer
    subgraph "Inference Layer"
        direction TB
        InferGen["inference/generator.py"]:::code
        InferRun["run_inference.py"]:::code
    end

    %% CLI Entry Point
    MainCLI["main.py (CLI Entry Point)"]:::code

    %% Exploratory Notebooks
    subgraph "Exploratory Notebooks"
        direction TB
        NB1["Mixture_of_Experts_from_Scratch.ipynb"]:::code
        NB2["Multi_Head_Latent_Attention_From_Scratch.ipynb"]:::code
        NB3["Multi_Token_Prediction_from_Scratch.ipynb"]:::code
    end

    %% External Services
    subgraph "External Services"
        direction TB
        HF["HuggingFace Datasets"]:::service
        Torch["PyTorch"]:::service
    end

    %% Data Flow Arrows
    RawData1 --> PrepFine
    RawData2 --> PrepTiny
    PrepFine --> DataLoader
    PrepTiny --> DataLoader
    HF --> DataLoader

    %% Model Instantiation
    MainCLI --> DataLoader
    MainCLI --> Trainer
    MainCLI --> InferRun

    DataLoader --> Trainer
    Config --> Trainer
    ModelAssembly --> Trainer
    MLA --> ModelAssembly
    MoE --> ModelAssembly
    MTP --> ModelAssembly
    Layers --> ModelAssembly

    Trainer --> Checkpoints
    Trainer --> WB
    Torch --> Trainer

    Checkpoints --> InferRun
    InferRun --> InferGen
    InferGen -->|"Generated Text"| MainCLI

    %% Notebooks usage
    NB1 --> MoE
    NB2 --> MLA
    NB3 --> MTP
    NB1 --> ModelAssembly
    NB2 --> ModelAssembly
    NB3 --> ModelAssembly
    NB1 --> DataLoader
    NB2 --> DataLoader
    NB3 --> DataLoader

    %% Click Events
    click PrepFine "https://github.com/work4ai/deepseek-from-scratch/blob/main/prepare_data_fineweb.py"
    click PrepTiny "https://github.com/work4ai/deepseek-from-scratch/blob/main/prepare_data_tiny_stories.py"
    click MainCLI "https://github.com/work4ai/deepseek-from-scratch/blob/main/main.py"
    click Config "https://github.com/work4ai/deepseek-from-scratch/blob/main/models/config.py"
    click ModelAssembly "https://github.com/work4ai/deepseek-from-scratch/blob/main/models/model.py"
    click MLA "https://github.com/work4ai/deepseek-from-scratch/blob/main/models/attention.py"
    click MoE "https://github.com/work4ai/deepseek-from-scratch/blob/main/models/moe.py"
    click MTP "https://github.com/work4ai/deepseek-from-scratch/blob/main/models/mtp.py"
    click Layers "https://github.com/work4ai/deepseek-from-scratch/blob/main/models/layers.py"
    click DataLoader "https://github.com/work4ai/deepseek-from-scratch/blob/main/training/data_loader.py"
    click Trainer "https://github.com/work4ai/deepseek-from-scratch/blob/main/training/trainer.py"
    click InferGen "https://github.com/work4ai/deepseek-from-scratch/blob/main/inference/generator.py"
    click InferRun "https://github.com/work4ai/deepseek-from-scratch/blob/main/run_inference.py"
    click NB1 "https://github.com/work4ai/deepseek-from-scratch/blob/main/notebooks/Mixture_of_Experts_from_Scratch.ipynb"
    click NB2 "https://github.com/work4ai/deepseek-from-scratch/blob/main/notebooks/Multi_Head_Latent_Attention_From_Scratch.ipynb"
    click NB3 "https://github.com/work4ai/deepseek-from-scratch/blob/main/notebooks/Multi_Token_Prediction_from_Scratch.ipynb"

    %% Styles
    classDef code fill:#D0E5FF,stroke:#004A99,stroke-width:1px
    classDef storage fill:#E0FFE0,stroke:#008000,stroke-width:1px
    classDef service fill:#FFE8C4,stroke:#FF8C00,stroke-width:1px
```
