# telefonica-scientific-research/gaze_reward architecture

> Architecture diagram of the GitHub repository telefonica-scientific-research/gaze_reward, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-03-26). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/telefonica-scientific-research/gaze_reward
- Repository: https://github.com/telefonica-scientific-research/gaze_reward

## Overview

##### What the project is
From the README and tree, this project implements:

- **Input data sources**: RLHF-style datasets such as **HelpSteer2** and **OpenAssistant**
- **Eye-tracking / gaze feature processing**: via `eyetrackpy` and `tokenizer_aligner`
- **Reward construction**: gaze-derived rewards for responses
- **Reward model variants**: `GazeAdd`, `GazeConcat`, soft-prompt variants, and general model variants
- **Training orchestration**: reward trainers that fine-tune LLMs
- **Support for multiple base LLMs**: Mistral and Llama family models
- **Local caching/storage**: LMDB-based storage for processed data

##### Entry point / orchestration

This is the **main control plane** for the project. It likely:
- parses CLI arguments
- selects dataset/model configuration
- chooses reward model and trainer
- launches training

##### External inputs
- **Hugging Face datasets**: `nvidia/HelpSteer2`, `OpenAssistant/oasst1`
- **Eye-tracking / gaze data**
- External packages:
- `tokenizer_aligner`
- `eyetrackpy`

These are not internal system modules but are critical dependencies and should be shown as **external sources/services**.

##### Data preprocessing and dataset processing
- `rlhf_rw/reward_utils/dataset_proceser_reward.py`
- `utils/dataset_proceser.py`

These likely handle:
- loading and normalizing raw dataset records
- aligning tokens with gaze spans
- preparing examples for reward computation
- converting raw inputs into training-ready samples

##### Local storage / caching

This suggests **LMDB-backed local persistence** for processed examples, cached features, or intermediate training artifacts.

##### Reward model layer
- `rlhf_rw/models/reward_model_base.py`
- `rlhf_rw/models/reward_model_factory.py`
- `rlhf_rw/models/reward_model_general_add.py`
- `rlhf_rw/models/reward_model_general_sp.py`

This is the heart of the project.

##### Training layer
- `rlhf_rw/trainers/reward_trainer.py`
- `rlhf_rw/trainers/reward_trainer_general.py`

These should represent the **optimization loop**:
- loading batches
- computing rewards
- backpropagation / parameter updates
- managing logging, scheduler, and training settings

##### CLI-driven pipeline
`main.py` is the entry point, so this is a **command-line controlled research pipeline**.

##### Factory pattern
`reward_model_factory.py` is a clear **factory pattern**.
This explicitly; it is one of the most important architecture signals in the tree.

##### Base class + concrete implementations
`reward_model_base.py` plus variant files strongly suggests **polymorphism / inheritance**.

##### Separation of concerns
The structure is already split into:
- `models/`
- `trainers/`
- `reward_utils/`
- `utils/`

##### Research pipeline, not runtime serving
There is **no API server, frontend, database service, or inference microservice** in the tree.

##### Layer 1: Inputs
Include:
- Hugging Face datasets
- raw gaze / eye-tracking data
- pretrained base LLMs
- external Python packages

##### Layer 2: Preprocessing
Include:
- dataset processors
- token/gaze alignment
- gaze feature extraction
- LMDB caching

##### Layer 3: Reward modeling
Include:
- reward model base class
- factory
- / SP variants

##### Layer 4: Training
Include:
- general trainer
- reward trainer
- training loop / optimization
- hyperparameters from CLI

##### Layer 5: Outputs
Include:
- trained reward-aligned LLM checkpoints
- logs / metrics
- evaluation results

##### dataset processor → LMDB storage
(if caching intermediate records)
##### trainer → checkpoints / evaluation metrics

- model selection
- dataset selection
- training hyperparameters
- feature flags like `--concat`, `--use_softprompt`, `--features_used`, `--fmv`

Those flags are important because they define the variant of the pipeline being executed.

##### Dataset branch
- `HelpSteer2`
- `OpenAssistant/oasst1`

##### Model branch
- `Mistral-7B-v0.3`
- `Llama-3-8B`

##### Reward method branch
- `GazeAdd`
- `GazeConcat`
- soft prompt / baseline variants

##### Feature selection branch
- `features_used` toggles which gaze signals are enabled
- `fp_dropout` affects feature projection regularization
- `fmv` selects feature model version

These branches should be shown as **decision points** or **configuration selectors**.

##### Technologies and libraries to mention
Even if not all are internal modules, they matter for the architecture:

- **Python 3.11.8**
- **PyTorch / Hugging Face ecosystem** likely underneath the training stack
- **tokenizer_aligner**
- **eyetrackpy**
- **LMDB**
- Hugging Face datasets/models

##### Reward Model Variants
- General Add
- General SP
##### Output: Fine-tuned Model + Metrics

- **gaze-driven reward construction**
- **LLM alignment**
- **variant-based reward modeling**
- **dataset/token/gaze alignment**
- **trainer-driven fine-tuning**

- **data acquisition**
- **feature engineering**
- **reward model selection**
- **training**
- **output/evaluation**

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% External Inputs
    subgraph EXT["External Inputs & Dependencies"]
        D1(["RLHF Datasets<br/>HelpSteer2<br/>OpenAssistant/oasst1"]):::external
        G1(["Raw Eye-Tracking / Gaze Data"]):::external
        B1(["Base LLM Backbones<br/>Mistral / Llama family"]):::external
        P1(["tokenizer_aligner"]):::external
        P2(["eyetrackpy"]):::external
        H1(["Hugging Face Hub / Datasets"]):::external
    end

    %% Orchestration
    subgraph ORCH["CLI Orchestration"]
        M0(["main.py<br/>CLI Entry Point"]):::orchestrator
    end

    %% Preprocessing
    subgraph PRE["Preprocessing & Alignment"]
        U1(["General Dataset Preprocessing<br/>Token-Gaze Alignment"]):::processing
        U2(["Reward Dataset Processing<br/>for Reward Construction"]):::processing
        C1[("LMDB Local Cache / Storage")]:::storage
    end

    %% Reward Modeling
    subgraph RM["Reward Modeling Layer"]
        R0(["Model Package Init"]):::model
        R1(["Reward Model Base"]):::model
        R2(["Reward Model Factory / Selector"]):::model
        R3(["GazeAdd Reward Model"]):::model
        R4(["GazeConcat / Soft-Prompt Reward Model"]):::model
    end

    %% Training
    subgraph TR["Training Orchestration"]
        T1(["General Reward Trainer"]):::trainer
        T2(["Reward Trainer"]):::trainer
    end

    %% Outputs
    subgraph OUT["Outputs"]
        O1(["Fine-tuned Reward-Aligned LLM"]):::output
        O2(["Training Logs / Metrics"]):::output
        O3(["Saved Checkpoints"]):::output
    end

    %% Core pipeline flow
    M0 -->|"selects dataset / model / flags"| D1
    M0 -->|"loads gaze inputs"| G1
    M0 -->|"configures base model"| B1
    M0 -->|"drives preprocessing"| U1
    M0 -->|"chooses reward variant"| R2
    M0 -->|"starts training"| T1

    D1 -->|"raw examples"| U1
    G1 -->|"gaze signals"| U1
    P1 -->|"token alignment support"| U1
    P2 -->|"gaze feature support"| U1
    H1 -->|"source registry"| D1

    U1 -->|"aligned samples"| U2
    U1 -->|"cache intermediate artifacts"| C1
    C1 -->|"reuse cached samples"| U2

    U2 -->|"training-ready reward inputs"| R2
    R0 -->|"exports abstractions"| R1
    R1 -->|"base interface"| R2
    R2 -->|"instantiates"| R3
    R2 -->|"instantiates"| R4

    R3 -->|"reward scores"| T2
    R4 -->|"reward scores"| T2
    T1 -->|"executes optimization"| T2

    B1 -->|"fine-tuning backbone"| T2
    T2 -->|"updates weights"| O1
    T2 -->|"writes metrics"| O2
    T2 -->|"stores snapshots"| O3

    %% Feedback / reuse
    O3 -.->|"restart / resume"| M0

    %% Click events
    click M0 "https://github.com/telefonica-scientific-research/gaze_reward/blob/main/rlhf_rw/main.py"
    click R1 "https://github.com/telefonica-scientific-research/gaze_reward/blob/main/rlhf_rw/models/reward_model_base.py"
    click R2 "https://github.com/telefonica-scientific-research/gaze_reward/blob/main/rlhf_rw/models/reward_model_factory.py"
    click R3 "https://github.com/telefonica-scientific-research/gaze_reward/blob/main/rlhf_rw/models/reward_model_general_add.py"
    click R4 "https://github.com/telefonica-scientific-research/gaze_reward/blob/main/rlhf_rw/models/reward_model_general_sp.py"
    click U2 "https://github.com/telefonica-scientific-research/gaze_reward/blob/main/rlhf_rw/reward_utils/dataset_proceser_reward.py"
    click U1 "https://github.com/telefonica-scientific-research/gaze_reward/blob/main/utils/dataset_proceser.py"
    click C1 "https://github.com/telefonica-scientific-research/gaze_reward/blob/main/utils/lmdb_storage.py"
    click T2 "https://github.com/telefonica-scientific-research/gaze_reward/blob/main/rlhf_rw/trainers/reward_trainer.py"
    click T1 "https://github.com/telefonica-scientific-research/gaze_reward/blob/main/rlhf_rw/trainers/reward_trainer_general.py"
    click R0 "https://github.com/telefonica-scientific-research/gaze_reward/blob/main/rlhf_rw/models/__init__.py"

    %% Styles
    classDef orchestrator fill:#dbeafe,stroke:#2563eb,stroke-width:2px,color:#0f172a;
    classDef processing fill:#dcfce7,stroke:#16a34a,stroke-width:2px,color:#052e16;
    classDef model fill:#ffedd5,stroke:#f97316,stroke-width:2px,color:#431407;
    classDef trainer fill:#ede9fe,stroke:#7c3aed,stroke-width:2px,color:#2e1065;
    classDef storage fill:#e2e8f0,stroke:#475569,stroke-width:2px,color:#0f172a;
    classDef external fill:#f8fafc,stroke:#94a3b8,stroke-dasharray: 5 5,color:#0f172a;
    classDef output fill:#ffe4e6,stroke:#e11d48,stroke-width:2px,color:#4c0519;
```
