How deepseek-ai/deepseek-coder works
This repository is a model application and evaluation toolkit for DeepSeek Coder, not the model pretraining code itself. The top-level architecture splits into three main concerns: benchmark evaluation, fine-tuning, and a small interactive demo. The README also documents how the released models are consumed through Hugging Face Transformers and vLLM.
1. Core boundary of the repo
The repo assumes pretrained DeepSeek Coder checkpoints are downloaded externally from Hugging Face. Most code here is about:
• loading those models,
• formatting prompts for different tasks,
• running generation,
• grading outputs,
• and comparing results across benchmarks.
2. Evaluation subsystem as the main product
The Evaluation/ directory is the largest part of the repo and is organized by benchmark family:
• HumanEval
• MBPP
• LeetCode
• DS-1000
• PAL-Math
Each benchmark folder contains its own dataset files, prompt/eval scripts, and helper code. The common architectural pattern is: dataset → prompt construction → model inference → execution/grading → metric report.
3. HumanEval and MBPP share a common execution model
Evaluation/HumanEval and Evaluation/MBPP both vendor a human_eval package with modules like data.py, execution.py, and evaluate_functional_correctness.py. This layer is the sandboxed code-execution core that:
• loads tasks from JSONL datasets,
• runs generated Python solutions,
• measures functional correctness using pass/fail outcomes,
• and computes benchmark metrics such as pass@k.
These folders also include eval_instruct.py, eval_pal.py, and eval.sh, showing support for multiple prompting styles:
• instruction-style generation,
• program-aided/“PAL” style reasoning,
• and shell-driven batch runs.
4. Benchmark-specific data and prompt layers
The benchmark folders keep their own data formats and configs close to the evaluator:
• HumanEval/data/* contains multilingual HumanEval variants.
• MBPP/data/* contains MBPP train/test JSONL files.
• test_config.yaml files define run settings for each benchmark.
This keeps benchmark semantics local, so the evaluator logic can stay generic while the prompts and test suites remain benchmark-specific.
5. LeetCode evaluation is a separate flow
Evaluation/LeetCode is structurally different from HumanEval/MBPP:
• evaluate_leetcode.py is the benchmark driver,
• vllm_inference.py indicates high-throughput batched inference via vLLM,
• and the local human_eval package provides shared execution/evaluation utilities.
This implies a flow optimized for larger-scale generation throughput, likely with multilingual task data in data/*.jsonl.
6. PAL-Math is a program-execution reasoning pipeline
Evaluation/PAL-Math is a distinct subsystem for math reasoning with executable intermediate steps. Its architecture is:
• task datasets under datasets/,
• prompt templates under prompts/,
• orchestration in run.py,
• grading/parsing/execution in utils/.
The important boundary here is that the model output is not only text; it is parsed into program-like reasoning, executed by python_executor.py, then checked by grader.py and parser.py. This makes PAL-Math an execution-backed reasoning benchmark rather than a pure text-generation benchmark.
7. Fine-tuning is isolated from evaluation
finetune/ is a separate training subsystem. It includes:
• finetune_deepseekcoder.py as the trainer entry point,
• a DeepSpeed ZeRO-3 config in configs/ds_config_zero3.json,
• and its own requirements file.
The data contract for fine-tuning is simple and externalized: instruction-following examples with instruction and output fields. The training path therefore is:
prepared instruction dataset → DeepSpeed training script → fine-tuned model checkpoint.
8. Inference modes are documented but not deeply integrated
The README describes three inference patterns that act like consumer interfaces to the model:
• plain completion with Transformers,
• chat completion using apply_chat_template,
• repository-level completion/infill,
• and high-throughput generation via vLLM.
Architecturally, these are not separate services in the repo; they are usage modes for the same underlying model checkpoints.
9. Demo is a thin interactive front end
demo/app.py plus style.css and requirement.txt form a small local UI layer. It likely wraps model inference behind a simple interactive interface for trying prompts and completions. This layer sits at the edge of the system and depends on the model runtime rather than owning any evaluation or training logic.
10. Shared utilities are duplicated per benchmark family
Several folders have utils/ or human_eval/ packages. These are not global abstractions; they are per-benchmark support libraries for:
• dataset loading,
• prompt formatting,
• code execution,
• grading,
• and result aggregation.
That duplication is intentional because each benchmark has slightly different task formats and scoring rules.
11. External runtime and infrastructure dependencies matter
The repo relies on a few major external technologies that shape the architecture:
• PyTorch / Transformers for model loading and generation,
• vLLM for scalable batched inference,
• DeepSpeed for distributed fine-tuning,
• and benchmark-specific execution of generated code in Python sandboxes.
These are not incidental libraries; they define the repo’s execution model and scalability boundaries.
12. Data assets are static and benchmark-local
The data/, datasets/, and prompts/ directories are mostly static assets. They are inputs to the evaluation pipeline, not runtime-generated outputs. The repo also includes model and code licenses separately, reinforcing the split between:
• repository code,
• benchmark assets,
• and external model weights/licenses.
13. How the pieces fit together
A good architecture diagram for this repo would show three parallel lanes:
1. Evaluation lane: benchmark data → prompt builder → model inference → code execution/grading → metrics.
2. Fine-tuning lane: instruction dataset → DeepSpeed trainer → adapted checkpoint.
3. Demo/inference lane: user prompt → Transformers or vLLM → generated completion/chat response.
The key architectural idea is that DeepSeek Coder is treated as an external pretrained model asset, while this repository provides the surrounding tooling to evaluate, adapt, and demo it across code and reasoning tasks.
Main entry points: Evaluation Eval hub · LeetCode LeetCode eval · run.py PAL-Math run · finetune_deepseekcoder.py Fine-tune script
Components 20
Evaluation
Benchmarking and scoring pipelines for code and math tasks.
- Eval hub · Top-level evaluation area that organizes benchmark-specific pipelines.
Evaluation - HumanEval data · Multilingual HumanEval task files and benchmark-local assets.
Evaluation/HumanEval/data - HumanEval runner · Instruction and PAL evaluation entrypoints plus shell batch execution.
Evaluation/HumanEval/eval_instruct.py - HumanEval core · Shared execution and correctness checking for generated programs.
Evaluation/HumanEval/human_eval - MBPP data · MBPP training and test JSONL data kept close to the evaluator.
Evaluation/MBPP/data - MBPP runner · Instruction and PAL evaluation entrypoints plus shell batch execution.
Evaluation/MBPP/eval_instruct.py - MBPP core · Shared execution and correctness checking for MBPP generations.
Evaluation/MBPP/human_eval - LeetCode eval · Separate evaluation flow optimized for larger-scale code generation.
Evaluation/LeetCode - PAL-Math assets · Math reasoning datasets and prompt templates.
Evaluation/PAL-Math/datasets - PAL-Math run · Main control flow for prompt execution and benchmark runs.
Evaluation/PAL-Math/run.py - PAL-Math exec · Parsing, execution, and grading of program-like model outputs.
Evaluation/PAL-Math/utils - DS-1000 · Standalone benchmark area documented separately from the code-heavy suites.
Evaluation/DS-1000
Fine-tuning
Instruction tuning pipeline for adapted checkpoints.
- Train data · External instruction-following examples with instruction/output fields.
- Fine-tune script · Main supervised fine-tuning entrypoint for DeepSeek Coder.
finetune/finetune_deepseekcoder.py - ZeRO-3 config · DeepSpeed configuration used to scale training.
finetune/configs/ds_config_zero3.json
Demo
Small interactive front end for trying the model.
- Interactive demo · Thin local front end for interactive prompt and completion testing.
demo/app.py
External runtime
Libraries and model assets used by the repo but not owned by it.
- DeepSeek Coder · Pretrained model weights consumed by the repo, downloaded separately.
- Transformers · Primary Hugging Face model loading and generation path.
- vLLM · High-throughput generation backend used by the LeetCode flow.
- DeepSpeed · Distributed fine-tuning backend, including ZeRO-3 support.
Connections 23
- DeepSeek Coder → Transformers: loads · Model weights are consumed through Hugging Face Transformers.
- DeepSeek Coder → vLLM: serves · The same checkpoints can be generated through vLLM.
- DeepSeek Coder → DeepSpeed: fine-tuned by · Training adapts the pretrained checkpoint using DeepSpeed.
- Eval hub → HumanEval data: uses · HumanEval evaluation starts from benchmark-local task files.
- Eval hub → HumanEval runner: runs · Runner scripts format prompts and launch evaluation jobs.
- HumanEval runner → HumanEval core: invokes · Generated solutions are checked through the shared execution core.
- HumanEval runner → Transformers: infers with · HumanEval uses standard model generation for completions.
- Eval hub → MBPP data: uses · MBPP evaluation starts from benchmark-local task files.
- Eval hub → MBPP runner: runs · Runner scripts format prompts and launch evaluation jobs.
- MBPP runner → MBPP core: invokes · Generated solutions are checked through the shared execution core.
- MBPP runner → Transformers: infers with · MBPP uses standard model generation for completions.
- LeetCode eval → vLLM: batched infer · LeetCode evaluation is optimized for high-throughput generation.
- LeetCode eval → Transformers: fallback infer · The benchmark flow still depends on model inference runtime.
- LeetCode eval → HumanEval core: reuses · LeetCode shares execution and evaluation utilities with code tasks.
- PAL-Math run → PAL-Math assets: loads · PAL-Math orchestrator consumes benchmark datasets and prompts.
- PAL-Math run → PAL-Math exec: executes · Program-like answers are parsed, executed, and graded.
- PAL-Math run → Transformers: queries · PAL-Math generates reasoning traces through the model runtime.
- Fine-tune script → Train data: trains on · Fine-tuning expects prepared instruction/output examples.
- Fine-tune script → ZeRO-3 config: configured by · Training scale and sharding are set by the ZeRO-3 config.
- Fine-tune script → DeepSpeed: uses · The trainer is built around DeepSpeed distributed training.
- Interactive demo → Transformers: queries · The demo is a thin UI over model inference.
- Interactive demo → DeepSeek Coder: shows · Interactive prompting uses the externally supplied checkpoint.
- Eval hub → DS-1000: documents · DS-1000 is present as a benchmark area with its own README.