# hendrycks/apps architecture

> Architecture diagram of the GitHub repository hendrycks/apps, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-01). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/hendrycks/apps
- Repository: https://github.com/hendrycks/apps

## Overview

##### Main Components
- Training Pipeline:
- Directory: train/
- Subcomponents include:
▸ Dataset Handling: “dataset_apps” and “dataset_lm” folders contain dataset utilities like APPSBaseDataset, base_lm_dataset, reindent scripts, etc.
▸ Model Training Utilities: Code for training such as dataset splitting (apps_create_split.py), callbacks (CustomTensorboardCallback.py), and tuning scripts (tune_apps_gpt.py).
▸ DeepSpeed Config: The JSON configuration (deepspeed_config.json) that sets up parameters for large-scale model training.
- Evaluation Pipeline:
- Directory: eval/
- Key elements include:
▸ Evaluation Scripts: eval_bleu.py, generate_gpt_codes.py, merge_codes.py, test_one_solution.py, etc.
▸ Job Scheduling: The “sbatch” folder holds SLURM scripts for job submission, indicating high-performance computing or batch processing.
▸ Utility Scripts: Scripts like “reindent.py” and “testing_util.py” for preprocessing and testing code solutions.
- Common/Shared Elements:
- Requirements file (requirements.txt) representing the dependency management.
- README files providing usage instructions.
- Additional support files (e.g., LICENSE, .gitignore).

##### Relationships and Interactions
- The training pipeline processes the APPS dataset to fine-tune the language models using training code and configuration files. There is a data flow from datasets (preprocessing, splitting) to model training, and subsequent monitoring via Tensorboard callbacks.
- The evaluation pipeline takes the output (i.e., the fine-tuned model weights and generated code) and runs various assessments. It integrates evaluation metrics (e.g., BLEU score calculation) and batch processing (via SLURM scripts) to systematically test the model on coding challenges.
- Even though the two pipelines operate largely independently, they are linked by the common objective of assessing coding challenge competence, and they both rely on standardized input data from the APPS dataset and shared pre/postprocessing utilities (like reindent scripts).

##### Architectural Patterns & Design Principles
- Separation of Concerns: Clear split between model training and evaluation which promotes modularity and reusability.
- Batch Processing & High-Performance Computing: Use of SLURM scripts in the evaluation directory indicates that the project supports distributed or batch processing tasks.
- Extensibility: The structure allows for additional dataset sources (e.g., dataset_lm for language modeling) and potential expansion of evaluation metrics.

##### Technologies, Frameworks, and Libraries
- Machine Learning Frameworks: Though not explicitly mentioned, the use of DeepSpeed configuration implies usage of PyTorch or similar frameworks.
- Job Scheduling: SLURM (through sbatch scripts) for submitting and managing evaluation tasks.
- Monitoring: Custom Tensorboard callback for tracking training metrics.

## Diagram (Mermaid)

```mermaid
flowchart TD
    RP["Research Project: Coding Challenge Competence"]:::research

    %% Training Pipeline Subgraph
    subgraph "Training Pipeline"
        T1A["APPS Dataset Handling"]:::training
        T1B["LM Dataset Handling"]:::training
        subgraph "Model Training Utilities"
            T2a["Data Split (apps_create_split.py)"]:::training
            T2b["Tensorboard Callback (CustomTensorboardCallback.py)"]:::training
            T2c["Tuning Script (tune_apps_gpt.py)"]:::training
            T2a --> T2b
            T2b --> T2c
        end
        T3["DeepSpeed Config (deepspeed_config.json)"]:::training
        T4["Fine-Tuned Model"]:::training

        T1A --> T2a
        T1B --> T2a
        T2c --> T3
        T3 --> T4
    end

    %% Evaluation Pipeline Subgraph
    subgraph "Evaluation Pipeline"
        E1a["Code Generation (generate_gpt_codes.py)"]:::evaluation
        E1b["Merge Codes (merge_codes.py)"]:::evaluation
        E1d["Test One Solution (test_one_solution.py)"]:::evaluation
        E1c["BLEU Evaluation (eval_bleu.py)"]:::evaluation
        E2["Job Scheduler (sbatch scripts)"]:::evaluation
        E3["Evaluation Utilities (reindent.py & testing_util.py)"]:::evaluation

        E1a --> E1b
        E1b --> E1d
        E1d --> E1c
        E1c --> E2
        E2 --> E3
    end

    %% Shared Components Subgraph
    subgraph "Shared Components"
        S1["requirements.txt"]:::shared
        S2["README.md"]:::shared
        S3["LICENSE"]:::shared
        S4[".gitignore"]:::shared
    end

    %% Connections from the Research Project node
    RP --> T1A
    RP --> T1B
    RP --> E1a
    RP --> S1

    %% Connection from Training to Evaluation (model production feeds evaluation)
    T4 --> E1a

    %% Click Events for Training Pipeline
    click T1A "https://github.com/hendrycks/apps/blob/main/train/dataset_apps/APPSBaseDataset.py"
    click T1B "https://github.com/hendrycks/apps/blob/main/train/dataset_lm/base_lm_dataset.py"
    click T2a "https://github.com/hendrycks/apps/blob/main/train/apps_create_split.py"
    click T2b "https://github.com/hendrycks/apps/blob/main/train/CustomTensorboardCallback.py"
    click T2c "https://github.com/hendrycks/apps/blob/main/train/tune_apps_gpt.py"
    click T3 "https://github.com/hendrycks/apps/blob/main/train/deepspeed_config.json"
    click T4 "https://github.com/hendrycks/apps/tree/main/train/"

    %% Click Events for Evaluation Pipeline
    click E1a "https://github.com/hendrycks/apps/blob/main/eval/generate_gpt_codes.py"
    click E1b "https://github.com/hendrycks/apps/blob/main/eval/merge_codes.py"
    click E1d "https://github.com/hendrycks/apps/blob/main/eval/test_one_solution.py"
    click E1c "https://github.com/hendrycks/apps/blob/main/eval/eval_bleu.py"
    click E2 "https://github.com/hendrycks/apps/tree/main/eval/sbatch/"
    click E3 "https://github.com/hendrycks/apps/blob/main/eval/reindent.py"

    %% Click Events for Shared Components
    click S1 "https://github.com/hendrycks/apps/blob/main/requirements.txt"
    click S2 "https://github.com/hendrycks/apps/blob/main/README.md"
    click S3 "https://github.com/hendrycks/apps/tree/main/LICENSE"
    click S4 "https://github.com/hendrycks/apps/blob/main/.gitignore"

    %% Styles
    classDef research fill:#ffb6c1,stroke:#000,stroke-width:1px;
    classDef training fill:#add8e6,stroke:#000,stroke-width:1px;
    classDef evaluation fill:#90ee90,stroke:#000,stroke-width:1px;
    classDef shared fill:#f0e68c,stroke:#000,stroke-width:1px;
```
