How telefonica-scientific-research/gaze_reward works
What the project is
From the README and tree, this project implements:
• Input data sources: RLHF-style datasets such as HelpSteer2 and OpenAssistant
• Eye-tracking / gaze feature processing: via eyetrackpy and tokenizer_aligner
• Reward construction: gaze-derived rewards for responses
• Reward model variants: GazeAdd, GazeConcat, soft-prompt variants, and general model variants
• Training orchestration: reward trainers that fine-tune LLMs
• Support for multiple base LLMs: Mistral and Llama family models
• Local caching/storage: LMDB-based storage for processed data
Entry point / orchestration
This is the main control plane for the project. It likely:
• parses CLI arguments
• selects dataset/model configuration
• chooses reward model and trainer
• launches training
External inputs
• Hugging Face datasets: nvidia/HelpSteer2, OpenAssistant/oasst1
• Eye-tracking / gaze data
• External packages:
• tokenizer_aligner
• eyetrackpy
These are not internal system modules but are critical dependencies and should be shown as external sources/services.
Data preprocessing and dataset processing
• rlhf_rw/reward_utils/dataset_proceser_reward.py
• utils/dataset_proceser.py
These likely handle:
• loading and normalizing raw dataset records
• aligning tokens with gaze spans
• preparing examples for reward computation
• converting raw inputs into training-ready samples
Local storage / caching
This suggests LMDB-backed local persistence for processed examples, cached features, or intermediate training artifacts.
Reward model layer
• rlhf_rw/models/reward_model_base.py
• rlhf_rw/models/reward_model_factory.py
• rlhf_rw/models/reward_model_general_add.py
• rlhf_rw/models/reward_model_general_sp.py
This is the heart of the project.
Training layer
• rlhf_rw/trainers/reward_trainer.py
• rlhf_rw/trainers/reward_trainer_general.py
These should represent the optimization loop:
• loading batches
• computing rewards
• backpropagation / parameter updates
• managing logging, scheduler, and training settings
CLI-driven pipeline
main.py is the entry point, so this is a command-line controlled research pipeline.
Factory pattern
reward_model_factory.py is a clear factory pattern.
This explicitly; it is one of the most important architecture signals in the tree.
Base class + concrete implementations
reward_model_base.py plus variant files strongly suggests polymorphism / inheritance.
Separation of concerns
The structure is already split into:
• models/
• trainers/
• reward_utils/
• utils/
Research pipeline, not runtime serving
There is no API server, frontend, database service, or inference microservice in the tree.
Layer 1: Inputs
Include:
• Hugging Face datasets
• raw gaze / eye-tracking data
• pretrained base LLMs
• external Python packages
Layer 2: Preprocessing
Include:
• dataset processors
• token/gaze alignment
• gaze feature extraction
• LMDB caching
Layer 3: Reward modeling
Include:
• reward model base class
• factory
• / SP variants
Layer 4: Training
Include:
• general trainer
• reward trainer
• training loop / optimization
• hyperparameters from CLI
Layer 5: Outputs
Include:
• trained reward-aligned LLM checkpoints
• logs / metrics
• evaluation results
dataset processor → LMDB storage
(if caching intermediate records)
trainer → checkpoints / evaluation metrics
• model selection
• dataset selection
• training hyperparameters
• feature flags like --concat, --use_softprompt, --features_used, --fmv
Those flags are important because they define the variant of the pipeline being executed.
Dataset branch
• HelpSteer2
• OpenAssistant/oasst1
Model branch
• Mistral-7B-v0.3
• Llama-3-8B
Reward method branch
• GazeAdd
• GazeConcat
• soft prompt / baseline variants
Feature selection branch
• features_used toggles which gaze signals are enabled
• fp_dropout affects feature projection regularization
• fmv selects feature model version
These branches should be shown as decision points or configuration selectors.
Technologies and libraries to mention
Even if not all are internal modules, they matter for the architecture:
• Python 3.11.8
• PyTorch / Hugging Face ecosystem likely underneath the training stack
• tokenizer_aligner
• eyetrackpy
• LMDB
• Hugging Face datasets/models
Reward Model Variants
• General Add
• General SP
Output: Fine-tuned Model + Metrics
• gaze-driven reward construction
• LLM alignment
• variant-based reward modeling
• dataset/token/gaze alignment
• trainer-driven fine-tuning
• data acquisition
• feature engineering
• reward model selection
• training
• output/evaluation