How servicenow/repofusion works
Project type and purpose
• This is an open-source research codebase (“RepoFusion”) that trains and evaluates repository-level code models.
• It contains two related subprojects:
a. Finetuning_CodeT5 – scripts to fine-tune CodeT5 models on code completion tasks
b. RepoFusion – scripts to train/evaluate a novel “RepoFusion” code model that incorporates repository contexts.
File structure
• Top level:
• Finetuning_CodeT5/ – config files, data prep, training, evaluation, utilities for CodeT5 fine-tuning
• RepoFusion/ – src/ (model, data, preprocessing, evaluation, options, Slurm support), train_reader.py, test_reader.py, notebooks, metrics
• README.md – project overview, dependencies, usage
• Patterns:
• A clear separation between “finetuning” and “main model training/eval” pipelines
• A data-loading + preprocessing stage, a model context module, a training module, and an evaluation/metrics module
• Configuration driven via YAML files in Finetuning_CodeT5/config
Key insights from README
• Dependencies: PyTorch, Transformers, Accelerate, Datasets, pandas, NumPy, tqdm
• Dataset: “Stack-Repo” – Java repos + three context types (PromptProposal, BM25, RandomNN)
• Training: torchrun on train_reader.py (reads contexts, initializes model, trains)
• Evaluation: torchrun on test_reader.py, then calculate_metrics.py to compute success rates
• Finetuned CodeT5 models are used as initialization for RepoFusion models
Main components & their relationships
A. Data Layer
• Input: Stack-Repo dataset (Java files + JSON context files)
• data.py / preprocess.py (parse JSON, load code files, build batches)
B. Configuration & Options
• YAML config files (Finetuning_CodeT5/config/*.yaml)
• options.py modules (RepoFusion/src/options.py & Finetuning_CodeT5/options.py)
• Control hyperparameters, dataset paths, model names, context settings
C. Model Context Module
• model_context.py (Finetuning_CodeT5) & parts of RepoFusion/src/model.py
• Constructs input sequences: code “holes” + repository contexts
D. Training Pipeline
• Finetuning_CodeT5/train.py & RepoFusion/train_reader.py
• Uses torchrun/Accelerate for distributed training (distributed.py in Finetuning)
• Initializes from pretrained or finetuned checkpoints → trains on batches
• Save checkpoints at intervals
E. Evaluation Pipeline
• Finetuning_CodeT5/evaluate.py & RepoFusion/test_reader.py
• Loads checkpoint, runs on eval split, produces raw predictions
F. Metrics & Reporting
• metrics.py (Finetuning_CodeT5) & calculate_metrics.py (RepoFusion)
• Compute completion success rates, summarizing model performance
G. Utilities
• util.py (common helper functions)
• slurm.py (RepoFusion/src/slurm.py) for cluster job submission
• notebooks (RepoFusion/calculate_metrics.ipynb)
Architectural patterns & design principles
• Modular, layered pipeline: Data → Preprocess → Context assembly → Model → Train/Eval → Metrics
• Configuration-driven design (YAML + options.py) for reproducibility
• Reuse of core modules (util, data) across both subprojects
• Separation of concerns: distinct scripts for training vs. evaluation vs. metrics
• Distributed training support via torchrun/Accelerate
Detail is key
• Script or module that plays a distinct role
• External dependencies and how they plug into the pipeline
• Both fine‐tuning and main training flows to reflect the full codebase