How deepseek-ai/esft works
ESFT specializes a sparse MoE language model by measuring expert activity, selecting task-relevant experts, fine-tuning those components, and evaluating the resulting adapter. Supporting capabilities include standard and expert-parallel training, single- and multi-GPU evaluation, and task-specific scoring for intent, law, summary, and translation. The source excerpts show the core orchestration and model adaptation; some subsystem links are documented in the README rather than verified in unsampled implementation details.
Main entry points: train.py Standard training · train_ep.py Expert-parallel training · eval.py Single-GPU evaluation · eval_multigpu.py Multi-GPU evaluation
Components 20
Expert analysis
- Training corpus
datasets/train - Expert scoring
scripts/expert/get_expert_scores.py - Expert score files
results/expert_scores - Expert selection
scripts/expert/generate_expert_config.py - Expert configuration
results/expert_configs
Specialized training
- Training preparation
utils.py - Standard training
train.py - Expert-parallel training
train_ep.py - ESFT adaptation
esft.py - Trained adapters
results
MoE model
- DeepSeek configuration
deepseek/configuration_deepseek.py - DeepSeek MoE model
deepseek/modeling_deepseek.py
Evaluation
- Evaluation datasets
datasets/eval - Task evaluators
benchmarks.py - Single-GPU evaluation
scripts/eval.py - Multi-GPU evaluation
eval_multigpu.py - Evaluation results
results/completions
Other components
- Researcher
- Pretrained model
- OpenAI GPT-4
Connections 32
- Researcher → Expert scoring: runs scoring
README.md - Training corpus → Expert scoring: samples tokens
scripts/expert/get_expert_scores.py - Expert scoring → Expert score files: logs expert use
scripts/expert/get_expert_scores.py - Expert score files → Expert selection: aggregates scores
scripts/expert/generate_expert_config.py - Expert selection → Expert configuration: writes selection
scripts/expert/generate_expert_config.py - Researcher → Standard training: runs training
README.md - Researcher → Expert-parallel training: runs parallel training
README.md - Expert configuration → Standard training: selects trainable experts
README.md - Expert configuration → Expert-parallel training: selects trainable experts
README.md - Training corpus → Training preparation: provides messages
- Training preparation → Standard training: formats examples
train.py - Training preparation → Expert-parallel training: formats examples
train_ep.py - Standard training → DeepSeek MoE model: loads model
train.py - Expert-parallel training → DeepSeek MoE model: loads model
train_ep.py - Standard training → ESFT adaptation: configures adaptation
train.py - Expert-parallel training → ESFT adaptation: configures adaptation
train_ep.py - Standard training → Trained adapters: saves checkpoints
README.md - Expert-parallel training → Trained adapters: saves checkpoints
README.md - DeepSeek MoE model → DeepSeek configuration: uses configuration
deepseek/modeling_deepseek.py - Researcher → Single-GPU evaluation: runs evaluation
README.md - Researcher → Multi-GPU evaluation: runs evaluation
README.md - Pretrained model → Single-GPU evaluation: loads base weights
scripts/eval.py - Pretrained model → Multi-GPU evaluation: loads base weights
eval_multigpu.py - Trained adapters → Single-GPU evaluation: loads adapter
scripts/eval.py - Trained adapters → Multi-GPU evaluation: loads adapter
eval_multigpu.py - Evaluation datasets → Single-GPU evaluation: supplies examples
scripts/eval.py - Evaluation datasets → Multi-GPU evaluation: supplies examples
eval_multigpu.py - Single-GPU evaluation → Task evaluators: evaluates predictions
scripts/eval.py - Multi-GPU evaluation → Task evaluators: evaluates predictions
eval_multigpu.py - Task evaluators → OpenAI GPT-4: requests GPT-4 scores
benchmarks.py - Single-GPU evaluation → Evaluation results: writes scores
scripts/eval.py - Multi-GPU evaluation → Evaluation results: writes rank results
eval_multigpu.py