# microsoft/codebert architecture

> Architecture diagram of the GitHub repository microsoft/codebert, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-12). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/microsoft/codebert
- Repository: https://github.com/microsoft/codebert

## Overview

##### Project Type and Purpose
- This is a monolithic repository of pretraining code and downstream evaluation for six code‐centric transformer models (CodeBERT, GraphCodeBERT, UniXcoder, CodeReviewer, CodeExecutor, LongCoder).
- It is not a web or full‐stack application but a research/ML pipeline framework.

##### File Structure
- Top-level directories:
- CodeBERT/
- GraphCodeBERT/
- UniXcoder/
- CodeExecutor/
- CodeReviewer/
- LongCoder/
- Each model directory contains subdirectories or files that implement:
- data processing (e.g., parser/DFG, dataset.py)
- model definitions (model.py)
- training or run scripts (run.py, run.sh)
- evaluation utilities (bleu.py, mrr.py, evaluator/)
- downstream tasks (in CodeExecutor and UniXcoder)

##### README Insights
- All six models share:
- A pretraining stage (mask/prediction, data‐flow injection, long‐range sequence handling, execution trace prediction).
- Downstream evaluation (code search, code generation, clone detection, code refinement, code review, code execution).
- Common dependencies: PyTorch, HuggingFace Transformers, shell scripts for orchestration.

##### Main Components
a. Data Layer
- Raw corpora (NL–PL pairs, code repositories, execution traces, change‐review data).
- Language‐specific data (Python, Java, etc.) and DFG parsers.
b. Preprocessing & Parsing
- Parser modules (e.g., parser/DFG build tools, utils).
- Dataset loaders per model (dataset.py).
c. Pretraining Engines
- CodeBERT, GraphCodeBERT, UniXcoder, LongCoder, CodeExecutor pretrain scripts and model classes.
- Shared transformer backbone (Roberta/Customized).
d. Model Artifacts
- Saved checkpoints (e.g., microsoft/codebert-base, codebert-base-mlm).
e. Downstream & Inference
- Code search, code summarization, clone detection, code review, code execution, refinement, translation.
- Separate “downstream” or “inference” subdirectories with run scripts and metrics.
f. Evaluation & Metrics
- BLEU, CodeBLEU, MRR, execution accuracy metrics.
- Utility modules (bleu.py, mrr.py, evaluator/).

##### Relationships & Data Flow
1) Raw Data → Parser/DFG → Tokenized/Graph-augmented Data
2) Data → Pretraining Module → Model Checkpoints
3) Checkpoints → Downstream Scripts → Inference Results
4) Inference Results → Metric Calculators → Performance Reports

##### Architectural Patterns & Design Principles
- Modular: Each model is isolated under its own directory.
- Layered Pipeline: clear separation into Data → Pretrain → Downstream → Evaluate.
- Reuse: Common patterns (parser build, run.sh orchestration, metrics) are replicated.
- Script-driven orchestration: shell wrappers call Python runs.

##### Technologies, Frameworks, Libraries
- Python 3, PyTorch, HuggingFace Transformers
- Shell scripts (bash) for trial orchestration
- Custom DFG parsers in C/Python (build.sh, utils.py)
- Evaluation tools: bleu.py, CodeBLEU suite

##### Detail is Key
- On the Data layer, list the various corpora (NL–PL pairs, code change diff, execution logs).
- On the Metrics layer, list the metric files (BLEU, MRR, CodeBLEU).

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data & Parsers
    subgraph "Data & Parsers"
        RawData1(("NL–PL Pairs\nCode Repositories\nExecution Logs")):::data
        Parser_CR_CB[/ "CodeReviewer CodeBLEU Parser" /]:::parser
        click Parser_CR_CB "https://github.com/microsoft/codebert/tree/master/CodeReviewer/code/evaluator/CodeBLEU/parser/"
        Parser_GCB_CD[/ "GraphCodeBERT CloneDetection Parser" /]:::parser
        click Parser_GCB_CD "https://github.com/microsoft/codebert/tree/master/GraphCodeBERT/clonedetection/parser/"
        Parser_GCB_CS[/ "GraphCodeBERT CodeSearch Parser" /]:::parser
        click Parser_GCB_CS "https://github.com/microsoft/codebert/tree/master/GraphCodeBERT/codesearch/parser/"
        Parser_GCB_RF[/ "GraphCodeBERT Refinement Parser" /]:::parser
        click Parser_GCB_RF "https://github.com/microsoft/codebert/tree/master/GraphCodeBERT/refinement/parser/"
        Parser_GCB_TR[/ "GraphCodeBERT Translation Parser" /]:::parser
        click Parser_GCB_TR "https://github.com/microsoft/codebert/tree/master/GraphCodeBERT/translation/parser/"
        Parser_LC[/ "LongCoder Parser" /]:::parser
        click Parser_LC "https://github.com/microsoft/codebert/tree/master/LongCoder/parser/"
    end

    %% Pretraining Stage
    subgraph "Pretraining Stage"
        CB["CodeBERT\n(model.py, run.sh)"]:::pretrain
        GCB["GraphCodeBERT\n(model.py, run.py)"]:::pretrain
        UX["UniXcoder\n(unixcoder.py)"]:::pretrain
        CE["CodeExecutor\n(model.py, run.sh)"]:::pretrain
        click CE "https://github.com/microsoft/codebert/blob/master/CodeExecutor/pretrain/model.py"
        LC["LongCoder\n(longcoder.py, model.py, run.sh)"]:::pretrain
        click LC "https://github.com/microsoft/codebert/blob/master/LongCoder/longcoder.py"
    end

    %% Model Checkpoints
    subgraph "Model Checkpoints"
        CP_CB(("CodeBERT Checkpoint")):::store
        CP_GCB(("GraphCodeBERT Checkpoint")):::store
        CP_UX(("UniXcoder Checkpoint")):::store
        CP_CE(("CodeExecutor Checkpoint")):::store
        CP_LC(("LongCoder Checkpoint")):::store
    end

    %% Downstream & Inference
    subgraph "Downstream & Inference"
        CE_DS["CodeExecutor Downstream"]:::downstream
        click CE_DS "https://github.com/microsoft/codebert/blob/master/CodeExecutor/downstream/run.py"
        CE_INF["CodeExecutor Inference"]:::downstream
        click CE_INF "https://github.com/microsoft/codebert/blob/master/CodeExecutor/inference/run.py"
        CR_DS["CodeReviewer Fine-tune & Inference"]:::downstream
        click CR_DS "https://github.com/microsoft/codebert/blob/master/CodeReviewer/code/run_*.py"
        GCB_CD["GraphCodeBERT Clone Detection"]:::downstream
        click GCB_CD "https://github.com/microsoft/codebert/blob/master/GraphCodeBERT/clonedetection/run.py"
        GCB_CS["GraphCodeBERT Code Search"]:::downstream
        click GCB_CS "https://github.com/microsoft/codebert/blob/master/GraphCodeBERT/codesearch/run.py"
        GCB_RF["GraphCodeBERT Refinement"]:::downstream
        click GCB_RF "https://github.com/microsoft/codebert/blob/master/GraphCodeBERT/refinement/run.py"
        GCB_TR["GraphCodeBERT Translation"]:::downstream
        click GCB_TR "https://github.com/microsoft/codebert/blob/master/GraphCodeBERT/translation/run.py"
        CB_CS["CodeBERT Code Search"]:::downstream
        click CB_CS "https://github.com/microsoft/codebert/blob/master/CodeBERT/codesearch/run_classifier.py"
        CB_CN["CodeBERT Code2NL"]:::downstream
        click CB_CN "https://github.com/microsoft/codebert/blob/master/CodeBERT/code2nl/run.py"
        UX_ZS["UniXcoder Zero-shot Search"]:::downstream
        click UX_ZS "https://github.com/microsoft/codebert/blob/master/UniXcoder/downstream-tasks/zero-shot-search/dataset/preprocess.py"
    end

    %% Metrics & Evaluation
    subgraph "Metrics & Evaluation"
        BLEU_CB["BLEU (CodeBERT)"]:::metrics
        click BLEU_CB "https://github.com/microsoft/codebert/blob/master/CodeBERT/code2nl/bleu.py"
        BLEU_CR["BLEU (CodeReviewer)"]:::metrics
        click BLEU_CR "https://github.com/microsoft/codebert/blob/master/CodeReviewer/code/evaluator/bleu.py"
        CBLEU_CR["CodeBLEU (CodeReviewer)"]:::metrics
        click CBLEU_CR "https://github.com/microsoft/codebert/blob/master/CodeReviewer/code/evaluator/CodeBLEU/bleu.py"
        MRR_CS["MRR (Code Search)"]:::metrics
        click MRR_CS "https://github.com/microsoft/codebert/blob/master/CodeBERT/codesearch/mrr.py"
        ACC_EX["Execution Accuracy"]:::metrics
        click ACC_EX "https://github.com/microsoft/codebert/blob/master/CodeExecutor/inference/metric.py"
    end

    %% Connections
    RawData1 -->|"raw data"| Parser_CR_CB
    RawData1 --> Parser_GCB_CD
    RawData1 --> Parser_GCB_CS
    RawData1 --> Parser_GCB_RF
    RawData1 --> Parser_GCB_TR
    RawData1 --> Parser_LC

    Parser_CR_CB -->|"tokenized+DFG"| CB
    Parser_GCB_CD --> GCB
    Parser_GCB_CS --> GCB
    Parser_GCB_RF --> GCB
    Parser_GCB_TR --> GCB
    Parser_LC --> LC

    CB -->|"model checkpoint"| CP_CB
    GCB -->|"model checkpoint"| CP_GCB
    UX -->|"model checkpoint"| CP_UX
    CE -->|"model checkpoint"| CP_CE
    LC -->|"model checkpoint"| CP_LC

    CP_CE -->|"checkpoint"| CE_DS
    CP_CE --> CE_INF
    CP_CB --> CB_CS
    CP_CB --> CB_CN
    CP_GCB --> GCB_CD
    CP_GCB --> GCB_CS
    CP_GCB --> GCB_RF
    CP_GCB --> GCB_TR
    CP_UX --> UX_ZS
    CP_CB --> CR_DS
    CP_GCB --> CR_DS
    CP_LC --> CR_DS

    CE_INF -->|"inference results"| ACC_EX
    CB_CN --> BLEU_CB
    CR_DS --> BLEU_CR
    CR_DS --> CBLEU_CR
    CB_CS --> MRR_CS

    classDef data fill:#ADD8E6,stroke:#333,stroke-width:1px;
    classDef parser fill:#B0C4DE,stroke:#333,stroke-width:1px;
    classDef pretrain fill:#90EE90,stroke:#333,stroke-width:1px;
    classDef store fill:#87CEFA,stroke:#333,stroke-width:2px;
    classDef downstream fill:#FFFACD,stroke:#333,stroke-width:1px;
    classDef metrics fill:#FFA500,stroke:#333,stroke-width:1px;
```
