How bailaiowo/deepseek-v4 works
This repository provides local inference for DeepSeek-V4, plus utilities to convert downloaded checkpoints into the inference format and to cast FP8 weights to BF16. The principal workflow is to load a model configuration, tokenizer, and model-parallel checkpoint, then generate token completions from interactive or batch prompts. The sampled sources show the generation entry point, model components, and conversion tools; README capabilities such as training and evaluation are described but have no corresponding runtime implementation in the supplied tree.
Main entry points: generate.py Generation entry · model.py Transformer block · fp8_cast_bf16.py FP8 weight caster · convert.py Checkpoint converter
Components 18
Text generation
- Generation entry
inference/generate.py - Tokenizer
- Model configs
inference/configs
Model computation
- Transformer
inference/model.py - Transformer block
inference/model.py - MLA attention
inference/model.py - Dense MLP
inference/model.py - MoE routing
inference/model.py - Expert networks
inference/model.py - Parallel layers
inference/model.py - FP8 kernels
inference/kernel.py
Checkpoint preparation
- Model-parallel weights
- Checkpoint converter
inference/convert.py - FP8 weight caster
inference/fp8_cast_bf16.py
Other components
- User or prompt file
- Hugging Face checkpoint
- FP8 weights
- BF16 weights
Connections 17
- User or prompt file → Generation entry: provides prompts
- Generation entry → Model configs: reads settings
- Generation entry → Tokenizer: encodes and decodes
- Generation entry → Transformer: runs inference
- Model-parallel weights → Generation entry: loads weights
- Transformer → Transformer block: applies layers
- Transformer block → MLA attention: computes attention
- Transformer block → Dense MLP: uses dense layers
- Transformer block → MoE routing: uses routed layers
- MoE routing → Expert networks: dispatches tokens
- MLA attention → Parallel layers: uses projections
- Dense MLP → Parallel layers: uses projections
- Hugging Face checkpoint → Checkpoint converter: reads weights
- Checkpoint converter → Model-parallel weights: writes shards
- FP8 weights → FP8 weight caster: reads weights
- FP8 weight caster → FP8 kernels: dequantizes with
- FP8 weight caster → BF16 weights: writes weights