# deepseek-ai/deepseek-coder-v2 architecture

> Architecture diagram of the GitHub repository deepseek-ai/deepseek-coder-v2, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-16). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/deepseek-ai/deepseek-coder-v2
- Repository: https://github.com/deepseek-ai/deepseek-coder-v2

## Overview

##### Project type and purpose
- This repository is the open-source release of a large mixture-of-experts (MoE) code language model—DeepSeek-Coder-V2.
- Its purpose is to provide pre-trained models (16B / 236B parameters), usage recipes for inference (Hugging Face Transformers, SGLang, vLLM), and API endpoints (DeepSeek Platform, chat website).
- There is no traditional “frontend/backend” code here; instead, the “components” are model artifacts, inference servers, deployment infrastructure, and client SDKs.

##### Main components to capture
A. Model Training Pipeline (offline)
– Data sources: Code corpora (6T tokens), math problems, benchmarks.
– Pre-processing & tokenization cluster.
– Distributed training cluster: GPU pods, parameter server, MoE expert layers.
B. Model Registry & Hosting
– Hugging Face Model Hub: artifact storage, versioning.
– DeepSeek Platform storage (S3 or internal).
C. Inference Platforms
1. Hugging Face Transformers
- Client SDK → HF Hub → Model loaded in PyTorch BF16 on GPU → local inference.
2. SGLang Server
- Launch server CLI → FP8/FP16 optimized inference → OpenAI-compatible REST API.
3. vLLM Server
- Custom vLLM build + Trust-Remote-Code → high-throughput inference → API.
4. DeepSeek Platform API
- API Gateway (REST) → Authentication/Rate Limiting → Inference cluster (model replicas) → Response.
5. Coder.deepseek.com Chat UI
- Browser UI → Web server (Next.js?) → Backend (Node/Python) → DeepSeek Platform API → Model inference → UI.
D. Client Integration
– Python scripts & notebooks (samples in README).
– OpenAI-compatible clients (openai-py).
– Third-party tools (Ollama, custom wrappers).

##### Relationships & data flows
- Training pipeline produces model artifacts → pushed to HF Hub and DeepSeek storage.
- Inference servers pull model artifacts on startup.
- Clients (Python SDK / Web UI) call inference servers via REST or WebSocket.
- API Gateway routes traffic to GPU inference pods.
- Responses return to clients.

##### Architectural patterns & design principles
- Microservices: Each inference engine (HF, SGLang, vLLM, DeepSeek API) is a separate service.
- Distributed Training: MoE experts distributed across GPU clusters.
- Model Registry: Abstraction between artifact storage and serving.
- API Compatibility: OpenAI-compatible REST interface for easy integration.
- Pluggable Inference Backends: Switchable between Transformers, SGLang, vLLM with minimal client changes.

##### Details
– Show number of model parameters next to each model artifact.
– Indicate context-length support (16K → 128K) near inference services.
– Mark “trust_remote_code” usage at the Transformers loader.
– Note that the API is compatible with OpenAI CLI & openai-py.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Training Pipeline
    subgraph "Training Pipeline"
        direction TB
        DataSources["Data Sources\n6T tokens, math, benchmarks"]:::blue
        Preproc["Pre-processing & Tokenization"]:::blue
        Trainer["Distributed Trainer\nGPU pods, Parameter Server\nMoE Expert layers"]:::blue
        Artifacts["Model Artifacts\n16B-Lite & 236B"]:::blue
    end

    %% Model Registry & Hosting
    subgraph "Model Registry & Hosting"
        direction TB
        HFHub["Hugging Face Model Hub"]:::storage
        DeepSeekStore["DeepSeek Platform Storage (S3)"]:::storage
    end

    %% Inference Platforms
    subgraph "Inference Platforms"
        direction TB
        HFserv["HF Transformers\nTorch BF16, trust_remote_code"]:::green
        SGLang["SGLang Server\nFP8/FP16, OpenAI-compatible REST"]:::green
        vLLM["vLLM Server\nHigh-throughput, Trust-Remote-Code\n(note: PR step)"]:::green
        DSPAPI["DeepSeek Platform API\nAPI Gateway (REST), Auth/Rate Limit"]:::green
        ChatUI["Coder.deepseek.com Chat UI"]:::green
    end

    %% Clients
    subgraph "Clients"
        direction TB
        PySDK["Python SDK & Notebooks"]:::orange
        OpenAIpy["openai-py & OpenAI CLI"]:::orange
        ThirdParty["Third-party Tools\n(Ollama, wrappers)"]:::orange
        Browser["Web Browser"]:::orange
    end

    %% Documentation
    subgraph "Documentation & Assets"
        direction TB
        README["Usage Recipes & Documentation"]:::gray
        Figures["Visual Assets for Diagrams"]:::gray
        Paper["Research Paper & Architecture Description"]:::gray
        LangList["Supported Languages List"]:::gray
    end

    %% Flows
    DataSources -->|preprocess| Preproc -->|train| Trainer -->|output artifacts| Artifacts
    Artifacts -->|push| HFHub
    Artifacts -->|push| DeepSeekStore

    HFHub -->|pull model| HFserv
    DeepSeekStore -->|pull model| SGLang
    DeepSeekStore -->|pull model| vLLM
    DeepSeekStore -->|pull model| DSPAPI

    PySDK -->|inference request/response| HFserv
    OpenAIpy -->|REST JSON| SGLang
    OpenAIpy -->|REST JSON| vLLM
    ThirdParty -->|inference| HFserv
    ThirdParty -->|inference| SGLang
    ThirdParty -->|inference| vLLM

    Browser -->|UI interaction| ChatUI
    ChatUI -->|REST JSON| DSPAPI
    DSPAPI -->|response| Browser

    %% Click Events
    click README "https://github.com/deepseek-ai/deepseek-coder-v2/blob/main/README.md"
    click Figures "https://github.com/deepseek-ai/deepseek-coder-v2/tree/main/figures/"
    click Paper "https://github.com/deepseek-ai/deepseek-coder-v2/blob/main/paper.pdf"
    click LangList "https://github.com/deepseek-ai/deepseek-coder-v2/blob/main/supported_langs.txt"

    %% Styles
    classDef blue fill:#cce5ff,stroke:#2f6f8f,color:#000;
    classDef green fill:#ccffcc,stroke:#2f8f2f,color:#000;
    classDef orange fill:#ffe0b3,stroke:#b36b00,color:#000;
    classDef gray fill:#f2f2f2,stroke:#666666,color:#000;
    classDef storage fill:#dddddd,stroke:#888888,color:#000;
```
