# servicenow/repofusion architecture

> Architecture diagram of the GitHub repository servicenow/repofusion, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-17). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/servicenow/repofusion
- Repository: https://github.com/servicenow/repofusion

## Overview

##### Project type and purpose
- This is an open-source research codebase (“RepoFusion”) that trains and evaluates repository-level code models.
- It contains two related subprojects:
a. Finetuning_CodeT5 – scripts to fine-tune CodeT5 models on code completion tasks
b. RepoFusion – scripts to train/evaluate a novel “RepoFusion” code model that incorporates repository contexts.

##### File structure
- Top level:
- Finetuning_CodeT5/ – config files, data prep, training, evaluation, utilities for CodeT5 fine-tuning
- RepoFusion/ – src/ (model, data, preprocessing, evaluation, options, Slurm support), train_reader.py, test_reader.py, notebooks, metrics
- README.md – project overview, dependencies, usage
- Patterns:
- A clear separation between “finetuning” and “main model training/eval” pipelines
- A data-loading + preprocessing stage, a model context module, a training module, and an evaluation/metrics module
- Configuration driven via YAML files in Finetuning_CodeT5/config

##### Key insights from README
- Dependencies: PyTorch, Transformers, Accelerate, Datasets, pandas, NumPy, tqdm
- Dataset: “Stack-Repo” – Java repos + three context types (PromptProposal, BM25, RandomNN)
- Training: torchrun on train_reader.py (reads contexts, initializes model, trains)
- Evaluation: torchrun on test_reader.py, then calculate_metrics.py to compute success rates
- Finetuned CodeT5 models are used as initialization for RepoFusion models

##### Main components & their relationships
A. Data Layer
- Input: Stack-Repo dataset (Java files + JSON context files)
- data.py / preprocess.py (parse JSON, load code files, build batches)
B. Configuration & Options
- YAML config files (Finetuning_CodeT5/config/*.yaml)
- options.py modules (RepoFusion/src/options.py & Finetuning_CodeT5/options.py)
- Control hyperparameters, dataset paths, model names, context settings
C. Model Context Module
- model_context.py (Finetuning_CodeT5) & parts of RepoFusion/src/model.py
- Constructs input sequences: code “holes” + repository contexts
D. Training Pipeline
- Finetuning_CodeT5/train.py & RepoFusion/train_reader.py
- Uses torchrun/Accelerate for distributed training (distributed.py in Finetuning)
- Initializes from pretrained or finetuned checkpoints → trains on batches
- Save checkpoints at intervals
E. Evaluation Pipeline
- Finetuning_CodeT5/evaluate.py & RepoFusion/test_reader.py
- Loads checkpoint, runs on eval split, produces raw predictions
F. Metrics & Reporting
- metrics.py (Finetuning_CodeT5) & calculate_metrics.py (RepoFusion)
- Compute completion success rates, summarizing model performance
G. Utilities
- util.py (common helper functions)
- slurm.py (RepoFusion/src/slurm.py) for cluster job submission
- notebooks (RepoFusion/calculate_metrics.ipynb)

##### Architectural patterns & design principles
- Modular, layered pipeline: Data → Preprocess → Context assembly → Model → Train/Eval → Metrics
- Configuration-driven design (YAML + options.py) for reproducibility
- Reuse of core modules (util, data) across both subprojects
- Separation of concerns: distinct scripts for training vs. evaluation vs. metrics
- Distributed training support via torchrun/Accelerate

##### Detail is key
- Script or module that plays a distinct role
- External dependencies and how they plug into the pipeline
- Both fine‐tuning and main training flows to reflect the full codebase

## Diagram (Mermaid)

```mermaid
graph TB
    %% External Dependencies
    subgraph "External Dependencies"
        Datasets["HuggingFace Datasets"]:::ext
        Transformers["Transformers"]:::ext
        PyTorch["PyTorch"]:::ext
        Accelerate["Accelerate"]:::ext
        SlurmExt["Slurm"]:::ext
    end

    %% Shared & Top-Level
    subgraph "Shared & Top-Level"
        README["README.md"]:::util
        LICENSE["LICENSE"]:::util
    end

    %% Finetune CodeT5 Subsystem
    subgraph "Finetuning_CodeT5 Subsystem"
        direction TB
        FT_Config["YAML Configs & Options"]:::config
        FT_Data["data.py / prep_data.py"]:::data
        FT_ModelCtx["model_context.py"]:::model
        FT_Train["train.py / distributed.py"]:::train
        FT_Eval["evaluate.py"]:::eval
        FT_Metrics["metrics.py"]:::eval
        FT_Util["util.py"]:::util
    end

    %% RepoFusion Subsystem
    subgraph "RepoFusion Subsystem"
        direction TB
        RF_Config["options.py"]:::config
        RF_Data["data.py / preprocess.py"]:::data
        RF_Model["model.py"]:::model
        RF_Train["train_reader.py"]:::train
        RF_Eval["test_reader.py / evaluation.py"]:::eval
        RF_Metrics["calculate_metrics.ipynb"]:::eval
        RF_Util["util.py"]:::util
        RF_Slurm["slurm.py"]:::util
    end

    %% Data Input
    DataInput["Stack-Repo JSON + .java files"]:::data

    %% Flows for Finetuning
    Datasets -->|provides dataset| FT_Data
    FT_Config -->|configures| FT_Data
    FT_Data -->|batches| FT_ModelCtx
    Transformers -->|pretrained models| FT_ModelCtx
    FT_ModelCtx -->|inputs| FT_Train
    PyTorch -->|training engine| FT_Train
    Accelerate -->|distributed launch| FT_Train
    FT_Train -->|checkpoint| FT_Eval
    FT_Eval -->|predictions| FT_Metrics
    FT_Util -->|helpers| FT_Data
    FT_Util -->|helpers| FT_Train

    %% Flows for RepoFusion
    Datasets -->|provides dataset| RF_Data
    RF_Config -->|configures| RF_Data
    RF_Data -->|batches| RF_Model
    Transformers -->|init weights| RF_Model
    RF_Model -->|inputs| RF_Train
    PyTorch -->|training engine| RF_Train
    Accelerate -->|distributed launch| RF_Train
    RF_Train -->|checkpoint| RF_Eval
    RF_Eval -->|predictions| RF_Metrics
    RF_Util -->|helpers| RF_Data
    RF_Util -->|helpers| RF_Train
    SlurmExt -->|job submission| RF_Slurm

    %% Shared & Top-Level usage
    README -->|overview| FT_Config
    README -->|overview| RF_Config

    %% Click Events for Finetuning
    click FT_Config "https://github.com/servicenow/repofusion/blob/main/Finetuning_CodeT5/config/base.yaml"
    click FT_Config "https://github.com/servicenow/repofusion/blob/main/Finetuning_CodeT5/config/finetuned_codet5base_512.yaml"
    click FT_Config "https://github.com/servicenow/repofusion/blob/main/Finetuning_CodeT5/config/finetuned_codet5large_512.yaml"
    click FT_Config "https://github.com/servicenow/repofusion/blob/main/Finetuning_CodeT5/options.py"
    click FT_Data "https://github.com/servicenow/repofusion/blob/main/Finetuning_CodeT5/data.py"
    click FT_Data "https://github.com/servicenow/repofusion/blob/main/Finetuning_CodeT5/prep_data.py"
    click FT_ModelCtx "https://github.com/servicenow/repofusion/blob/main/Finetuning_CodeT5/model_context.py"
    click FT_Train "https://github.com/servicenow/repofusion/blob/main/Finetuning_CodeT5/train.py"
    click FT_Train "https://github.com/servicenow/repofusion/blob/main/Finetuning_CodeT5/distributed.py"
    click FT_Eval "https://github.com/servicenow/repofusion/blob/main/Finetuning_CodeT5/evaluate.py"
    click FT_Metrics "https://github.com/servicenow/repofusion/blob/main/Finetuning_CodeT5/metrics.py"
    click FT_Util "https://github.com/servicenow/repofusion/blob/main/Finetuning_CodeT5/util.py"

    %% Click Events for RepoFusion
    click RF_Config "https://github.com/servicenow/repofusion/blob/main/RepoFusion/src/options.py"
    click RF_Data "https://github.com/servicenow/repofusion/blob/main/RepoFusion/src/data.py"
    click RF_Data "https://github.com/servicenow/repofusion/blob/main/RepoFusion/src/preprocess.py"
    click RF_Model "https://github.com/servicenow/repofusion/blob/main/RepoFusion/src/model.py"
    click RF_Train "https://github.com/servicenow/repofusion/blob/main/RepoFusion/train_reader.py"
    click RF_Eval "https://github.com/servicenow/repofusion/blob/main/RepoFusion/test_reader.py"
    click RF_Eval "https://github.com/servicenow/repofusion/blob/main/RepoFusion/src/evaluation.py"
    click RF_Metrics "https://github.com/servicenow/repofusion/blob/main/RepoFusion/calculate_metrics.ipynb"
    click RF_Util "https://github.com/servicenow/repofusion/blob/main/RepoFusion/src/util.py"
    click RF_Slurm "https://github.com/servicenow/repofusion/blob/main/RepoFusion/src/slurm.py"

    %% Click Events for Shared
    click README "https://github.com/servicenow/repofusion/blob/main/README.md"
    click LICENSE "https://github.com/servicenow/repofusion/tree/main/LICENSE"

    %% Styles
    classDef data fill:#D0E8FF,stroke:#005B96,color:#000;
    classDef model fill:#DFF2BF,stroke:#4F8A10,color:#000;
    classDef train fill:#BCF5A9,stroke:#4E9A06,color:#000;
    classDef eval fill:#FEEFB3,stroke:#9F6000,color:#000;
    classDef config fill:#ECECEC,stroke:#888888,color:#000;
    classDef util fill:#EDEDED,stroke:#666666,color:#000;
    classDef ext fill:#F5EEEE,stroke:#AA0000,color:#000;
```
