# damanimehul/rlcr architecture

> Architecture diagram of the GitHub repository damanimehul/rlcr, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-10-23). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/damanimehul/rlcr
- Repository: https://github.com/damanimehul/rlcr

## Overview

##### Project Type and Purpose
- Project type: Research‐oriented ML pipeline implementing reinforcement learning for large‐language models.
- Purpose: Train LMs with uncertainty‐aware reward signals (Beyond Binary Rewards) and evaluate them across QA/math datasets.

##### File Structure
- Top‐level scripts:
– rl_runner.py, GRPO_Trainer.py: Orchestrate training with TRL + deepspeed/accelerate
– evaluation.py, inference_example.py: Run batch evaluation and demo inference
– train_runs.sh, eval_runs.sh: Convenience wrappers
– dataset_processing.py: Dataset loading & preprocessing
– reward_fns.py: Custom reward‐computation functions
- Directories:
– configs/Qwen-7B/*: YAML training configs (datasets, RLCR/RLVR variants)
– data/*: Raw & processed dataset splits + creation scripts
– eval/: Core evaluation functions & utils
– eval_configs/*: JSON eval scenario definitions
– eval_outputs/ & results/: Persisted eval outputs & metrics
- Config/build files: deepspeed.yaml, environment.yml

##### Extract README Insights
- Key external dependencies: HuggingFace TRL, Open-R1, deepspeed, accelerate, wandb
- Training flow: accelerate launch → rl_runner → distributed RL training on multi‐GPU
- Evaluation flow: evaluation.py / eval_runs.sh → generate predictions → compute metrics

##### Main Components & Their Interactions
a. Configuration Manager
– Inputs: YAML (configs/) for training, JSON (eval_configs/) for evaluation
– Provides hyperparameters, model paths, dataset identifiers
b. Data Ingestion & Preprocessing
– Supplies batched examples to the training or evaluation pipeline
c. Training Orchestrator
– rl_runner.py and GRPO_Trainer.py wrap HuggingFace TRL RL loops
– Leverages reward_fns.py for computing sample‐level rewards
– Uses deepspeed.yaml + accelerate for distributed execution
– Logs metrics and intermediate generations to wandb
d. Reward Module
– reward_fns.py encapsulates “Beyond Binary” reward logic
– Called by the RL loop to score model outputs against ground truth
e. Model Artifact Manager
– Checkpoints saved locally or to HuggingFace hub
– Config files (configs/Qwen-7B/…) track experiment provenance
f. Evaluation Pipeline
– evaluation.py invokes eval/check_functions.py, eval_utils.py
– Consumes eval_configs/*, loads trained model, runs inference
– Writes raw outputs to eval_outputs/ and summary metrics to results/
g. Inference Demo
– inference_example.py: spin up single‐GPU run for ad‐hoc testing
h. Auxiliary Scripts
– train_runs.sh / eval_runs.sh: Shell scripts combining steps above

##### Architectural Patterns & Principles
- Modular ML‐pipeline: Clear separation of “Config → Data → Training → Reward → Logging → Evaluation”
- Plugin‐style Reward & Eval: New datasets or reward strategies can be added by dropping files under data/creation_scripts/, reward_fns.py, or eval_configs/
- Distributed Compute Abstraction: Deepspeed + Accelerate hide multi‐GPU details
- CLI‐driven Orchestration: Single‐entrypoints (rl_runner.py, evaluation.py, inference_example.py)

##### How to Represent This
– Configuration (YAML/JSON)
– Data storage (data/, eval_outputs/)
– Compute modules (Training, Reward, Evaluation)
– External services (wandb, HuggingFace Model Hub, GPUs)
– Config → Runner → Data Loader → RL Loop → Reward Module → Checkpoints → Evaluation → Results
– Runner → wandb API calls
- Critical dependencies:
– TRL & Open-R1 libraries under “RL Loop”
– deepspeed & accelerate under “Distributed Compute”
– datasets & transformers under “Data Loader” and “Model”

## Diagram (Mermaid)

```mermaid
flowchart TD
    subgraph "Configuration Manager"
        CFGT["YAML Configs\n(configs/Qwen-7B/)"]:::data
        CFGE["JSON Eval Configs\n(eval_configs/)"]:::data
    end

    subgraph "Data Ingestion & Preprocessing"
        DATA["Raw & Processed Data\n(data/, creation_scripts/)"]:::data
        DI["Data Loader\n(dataset_processing.py)"]:::code
    end

    subgraph "Training Pipeline"
        RT["RL Trainer\n(rl_runner.py / GRPO_Trainer.py)"]:::code
        RE["Reward Engine\n(reward_fns.py)"]:::code
        DC["Distributed Compute\n(deepspeed & accelerate)"]:::compute
        LA["Logging\n(wandb)"]:::external
        MA["Model Artifact Manager"]:::data
    end

    subgraph "Evaluation Pipeline"
        EP["Evaluation Engine\n(evaluation.py)"]:::code
        ECF["Check Functions\n(eval/check_functions.py)"]:::code
        EU["Eval Utils\n(eval/eval_utils.py)"]:::code
        EA["Eval Args\n(eval/eval_args.py)"]:::code
        EO["Eval Outputs\n(eval_outputs/)"]:::data
        RES["Results\n(results/)"]:::data
    end

    subgraph "Inference Demo"
        Inf["Inference Demo\n(inference_example.py)"]:::code
    end

    subgraph "Auxiliary Scripts"
        ARUN["CLI Wrappers\n(train_runs.sh / eval_runs.sh)"]:::code
    end

    Env["Env & Build\n(environment.yml)"]:::code
    HFHUB["HuggingFace Hub"]:::external
    GPUs["GPUs"]:::compute

    %% Flows
    CFGT --> DI
    CFGT --> RT
    CFGE --> EP
    DATA --> DI
    DI --> RT
    DI --> EP
    DI --> Inf

    RT --> DC
    DC --> RT
    RT --> RE
    RE --> RT
    RT --> LA
    RT --> MA
    MA --> HFHUB

    RT --> GPUs
    DC --> GPUs

    EP --> ECF
    EP --> EU
    EP --> EA
    EP --> GPUs
    EP --> EO
    EP --> RES

    Inf --> GPUs

    ARUN --> RT
    ARUN --> EP
    ARUN --> Inf

    Env --> DC
    Env --> EP
    Env --> Inf

    %% Click Events
    click CFGT "https://github.com/damanimehul/rlcr/tree/main/configs/Qwen-7B/"
    click CFGE "https://github.com/damanimehul/rlcr/tree/main/eval_configs/"
    click DATA "https://github.com/damanimehul/rlcr/tree/main/data/creation_scripts/"
    click DI "https://github.com/damanimehul/rlcr/blob/main/dataset_processing.py"
    click RT "https://github.com/damanimehul/rlcr/blob/main/rl_runner.py"
    click RT "https://github.com/damanimehul/rlcr/blob/main/GRPO_Trainer.py"
    click RT "https://github.com/damanimehul/rlcr/blob/main/trainer_utils.py"
    click RT "https://github.com/damanimehul/rlcr/blob/main/arguments.py"
    click DC "https://github.com/damanimehul/rlcr/blob/main/deepspeed.yaml"
    click RE "https://github.com/damanimehul/rlcr/blob/main/reward_fns.py"
    click MA "https://github.com/damanimehul/rlcr/blob/main/configs/Qwen-7B/*/*.yaml"
    click EP "https://github.com/damanimehul/rlcr/blob/main/evaluation.py"
    click ECF "https://github.com/damanimehul/rlcr/blob/main/eval/check_functions.py"
    click EU "https://github.com/damanimehul/rlcr/blob/main/eval/eval_utils.py"
    click EA "https://github.com/damanimehul/rlcr/blob/main/eval/eval_args.py"
    click EO "https://github.com/damanimehul/rlcr/tree/main/eval_outputs/"
    click RES "https://github.com/damanimehul/rlcr/tree/main/results/"
    click Inf "https://github.com/damanimehul/rlcr/blob/main/inference_example.py"
    click ARUN "https://github.com/damanimehul/rlcr/blob/main/train_runs.sh"
    click ARUN "https://github.com/damanimehul/rlcr/blob/main/eval_runs.sh"
    click Env "https://github.com/damanimehul/rlcr/blob/main/environment.yml"

    %% Styles
    classDef code fill:#D0E6FF,stroke:#6CA0DC,color:#034A8F
    classDef data fill:#E6F4EA,stroke:#48A14D,color:#0B6623
    classDef external fill:#FFE5CC,stroke:#FFA14F,color:#A65E2E
    classDef compute fill:#E8E0FF,stroke:#B39DDB,color:#4B0082
```
