# bailaiowo/deepseek-v4 architecture

> Architecture diagram of the GitHub repository bailaiowo/deepseek-v4, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/bailaiowo/deepseek-v4
- Repository: https://github.com/bailaiowo/deepseek-v4

## Overview

This repository provides local inference for DeepSeek-V4, plus utilities to convert downloaded checkpoints into the inference format and to cast FP8 weights to BF16. The principal workflow is to load a model configuration, tokenizer, and model-parallel checkpoint, then generate token completions from interactive or batch prompts. The sampled sources show the generation entry point, model components, and conversion tools; README capabilities such as training and evaluation are described but have no corresponding runtime implementation in the supplied tree.

Main entry points: [`inference/generate.py`](https://github.com/bailaiowo/deepseek-v4/blob/main/inference/generate.py) (Generation entry), [`inference/model.py`](https://github.com/bailaiowo/deepseek-v4/blob/main/inference/model.py) (Transformer block), [`inference/fp8_cast_bf16.py`](https://github.com/bailaiowo/deepseek-v4/blob/main/inference/fp8_cast_bf16.py) (FP8 weight caster), [`inference/convert.py`](https://github.com/bailaiowo/deepseek-v4/blob/main/inference/convert.py) (Checkpoint converter)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_generation["Text generation"]
  node_generate["Generation entry<br/>[generate.py]"]
  node_tokenizer["Tokenizer"]
  node_config["Model configs"]
end

subgraph group_model["Model computation"]
  node_transformer["Transformer<br/>[model.py]"]
  node_block["Transformer block<br/>[model.py]"]
  node_mla["MLA attention<br/>[model.py]"]
  node_ffn["Dense MLP<br/>[model.py]"]
  node_moe["MoE routing<br/>[model.py]"]
  node_expert["Expert networks<br/>[model.py]"]
  node_parallel["Parallel layers<br/>[model.py]"]
  node_fp8kernel["FP8 kernels<br/>[kernel.py]"]
end

subgraph group_weights["Checkpoint preparation"]
  node_converted["Model-parallel weights"]
  node_convert["Checkpoint converter<br/>[convert.py]"]
  node_fp8cast["FP8 weight caster<br/>[fp8_cast_bf16.py]"]
end

node_user(("User or prompt file"))
node_hfcheckpoint["Hugging Face checkpoint"]
node_fp8weights["FP8 weights"]
node_bf16weights["BF16 weights"]

node_user -->|"provides prompts"| node_generate
node_generate -->|"reads settings"| node_config
node_generate -->|"encodes and decodes"| node_tokenizer
node_generate -->|"runs inference"| node_transformer
node_converted -->|"loads weights"| node_generate
node_transformer -->|"applies layers"| node_block
node_block -->|"computes attention"| node_mla
node_block -->|"uses dense layers"| node_ffn
node_block -->|"uses routed layers"| node_moe
node_moe -->|"dispatches tokens"| node_expert
node_mla -->|"uses projections"| node_parallel
node_ffn -->|"uses projections"| node_parallel
node_hfcheckpoint -->|"reads weights"| node_convert
node_convert -->|"writes shards"| node_converted
node_fp8weights -->|"reads weights"| node_fp8cast
node_fp8cast -->|"dequantizes with"| node_fp8kernel
node_fp8cast -->|"writes weights"| node_bf16weights

click node_generate "https://github.com/bailaiowo/deepseek-v4/blob/main/inference/generate.py"
click node_config "https://github.com/bailaiowo/deepseek-v4/tree/main/inference/configs"
click node_transformer "https://github.com/bailaiowo/deepseek-v4/blob/main/inference/model.py"
click node_block "https://github.com/bailaiowo/deepseek-v4/blob/main/inference/model.py"
click node_mla "https://github.com/bailaiowo/deepseek-v4/blob/main/inference/model.py"
click node_ffn "https://github.com/bailaiowo/deepseek-v4/blob/main/inference/model.py"
click node_moe "https://github.com/bailaiowo/deepseek-v4/blob/main/inference/model.py"
click node_expert "https://github.com/bailaiowo/deepseek-v4/blob/main/inference/model.py"
click node_parallel "https://github.com/bailaiowo/deepseek-v4/blob/main/inference/model.py"
click node_fp8kernel "https://github.com/bailaiowo/deepseek-v4/blob/main/inference/kernel.py"
click node_convert "https://github.com/bailaiowo/deepseek-v4/blob/main/inference/convert.py"
click node_fp8cast "https://github.com/bailaiowo/deepseek-v4/blob/main/inference/fp8_cast_bf16.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_generate,node_tokenizer,node_config,node_user toneBlue
class node_transformer,node_block,node_mla,node_ffn,node_moe,node_expert,node_parallel,node_fp8kernel toneAmber
class node_converted,node_convert,node_fp8cast toneMint
class node_hfcheckpoint,node_fp8weights,node_bf16weights toneIndigo
```

## Components

### Text generation

- [Generation entry](https://github.com/bailaiowo/deepseek-v4/blob/main/inference/generate.py): `inference/generate.py`
- **Tokenizer**
- [Model configs](https://github.com/bailaiowo/deepseek-v4/tree/main/inference/configs): `inference/configs`

### Model computation

- [Transformer](https://github.com/bailaiowo/deepseek-v4/blob/main/inference/model.py): `inference/model.py`
- [Transformer block](https://github.com/bailaiowo/deepseek-v4/blob/main/inference/model.py): `inference/model.py`
- [MLA attention](https://github.com/bailaiowo/deepseek-v4/blob/main/inference/model.py): `inference/model.py`
- [Dense MLP](https://github.com/bailaiowo/deepseek-v4/blob/main/inference/model.py): `inference/model.py`
- [MoE routing](https://github.com/bailaiowo/deepseek-v4/blob/main/inference/model.py): `inference/model.py`
- [Expert networks](https://github.com/bailaiowo/deepseek-v4/blob/main/inference/model.py): `inference/model.py`
- [Parallel layers](https://github.com/bailaiowo/deepseek-v4/blob/main/inference/model.py): `inference/model.py`
- [FP8 kernels](https://github.com/bailaiowo/deepseek-v4/blob/main/inference/kernel.py): `inference/kernel.py`

### Checkpoint preparation

- **Model-parallel weights**
- [Checkpoint converter](https://github.com/bailaiowo/deepseek-v4/blob/main/inference/convert.py): `inference/convert.py`
- [FP8 weight caster](https://github.com/bailaiowo/deepseek-v4/blob/main/inference/fp8_cast_bf16.py): `inference/fp8_cast_bf16.py`

### Other components

- **User or prompt file**
- **Hugging Face checkpoint**
- **FP8 weights**
- **BF16 weights**

## Connections

- User or prompt file → Generation entry: provides prompts
- Generation entry → Model configs: reads settings
- Generation entry → Tokenizer: encodes and decodes
- Generation entry → Transformer: runs inference
- Model-parallel weights → Generation entry: loads weights
- Transformer → Transformer block: applies layers
- Transformer block → MLA attention: computes attention
- Transformer block → Dense MLP: uses dense layers
- Transformer block → MoE routing: uses routed layers
- MoE routing → Expert networks: dispatches tokens
- MLA attention → Parallel layers: uses projections
- Dense MLP → Parallel layers: uses projections
- Hugging Face checkpoint → Checkpoint converter: reads weights
- Checkpoint converter → Model-parallel weights: writes shards
- FP8 weights → FP8 weight caster: reads weights
- FP8 weight caster → FP8 kernels: dequantizes with
- FP8 weight caster → BF16 weights: writes weights