How deepseek-ai/deepseek-v3 works
This repository provides a local inference and checkpoint-preparation toolkit for DeepSeek-V3. The main workflow loads a configured Transformer and checkpoint, tokenizes interactive or batch prompts, generates and decodes completions, and returns them to the user. Separate utilities shard Hugging Face checkpoints for model-parallel inference and convert FP8 weights to BF16. The excerpts show the generator-to-model and conversion-to-kernel dependencies; other model internals and README-described capabilities are represented at a higher level where implementation details were not sampled.
Main entry points: generate.py Text generation · fp8_cast_bf16.py FP8 to BF16 · model.py Transformer model · convert.py Checkpoint sharding
Components 12
Inference runtime
- Text generation
inference/generate.py - Tokenizer
- Model configs
inference/configs
Model execution
- Transformer model
inference/model.py - Triton kernels
inference/kernel.py
Weight preparation
- Checkpoint sharding
inference/convert.py - FP8 to BF16
inference/fp8_cast_bf16.py
Other components
- Prompt user
- Model checkpoint
- Generated text
- Hugging Face weights
- Prepared weights
Connections 13
- Prompt user → Text generation: submits prompts
inference/generate.py - Text generation → Model configs: reads settings
inference/generate.py - Text generation → Tokenizer: tokenizes prompts
inference/generate.py - Text generation → Model checkpoint: loads weights
inference/generate.py - Text generation → Transformer model: runs inference
inference/generate.py - Transformer model → Triton kernels: uses kernels
inference/model.py - Text generation → Generated text: produces text
inference/generate.py - Generated text → Prompt user: returns result
- Hugging Face weights → Checkpoint sharding: provides weights
inference/convert.py - Checkpoint sharding → Prepared weights: writes shards
inference/convert.py - Hugging Face weights → FP8 to BF16: provides FP8 weights
inference/fp8_cast_bf16.py - FP8 to BF16 → Triton kernels: dequantizes with
inference/fp8_cast_bf16.py - FP8 to BF16 → Prepared weights: writes BF16 weights
inference/fp8_cast_bf16.py