How work4ai/deepseek-from-scratch works
Project type and purpose
• This is an open-source machine-learning project that implements a transformer-based language model (DeepSeek V3) from scratch, complete with training and inference pipelines.
• Its purpose is to demonstrate and experiment with innovations (MLA, MoE, MTP) on top of a standard transformer, train on FineWeb-Edu, and serve text generation.
File structure
• Top-level scripts and config
– main.py: CLI entry-point for training or other modes
– run_inference.py: standalone inference runner
– prepare_data_fineweb.py / prepare_data_tiny_stories.py: data-prep scripts
– requirements.txt, .gitignore, README.md
• models/
– attention.py, moe.py, mtp.py: core algorithm modules
– layers.py: RoPE, RMSNorm, SwiGLU implementations
– model.py: assembles transformer blocks
– config.py: hyperparameters and architecture settings
• training/
– data_loader.py: dataset reading, tokenization, batching
– trainer.py: training loop, optimizer, learning rate schedulers, W&B logging
• inference/
– generator.py: sampling/speculative decoding logic
– run_inference.py: loads checkpoint, runs generator on prompts
• notebooks/: Jupyter demonstrations of each innovation
README for additional insights
• Highlights external dependencies: HuggingFace datasets, Weights & Biases for logging, PyTorch (implied).
• Shows model parameter counts, dataset sizes, training metrics.