How modalminds/mm-eureka works
Main Components
a. Repository/Infrastructure:
• • • • • •
• GitHub workflows, Dockerfiles, and examples indicate that the project supports CI/CD and containerized deployment as well as running training scripts on both single-node and multi-node setups.
b. Project Core Library (openrlhf Package):
• CLI Tools: Located in openrlhf/cli, these provide entry points for interactive chat, batch inference, and various training commands (e.g., train_sft, train_ppo, train_dpo, etc.).
• Datasets: Under openrlhf/datasets, there are processes to prepare, filter, and structure multiple types of data (reward datasets, SFT datasets, unpaired preference datasets, etc.) used for training.
• Models: The openrlhf/models directory contains model definitions, including the base model and specialized modules (under lmm_kits) for both the common functionality and specific adaptations (like Qwen2.5-VL). These are tightly coupled with processing and patching functions.
• Trainers: In openrlhf/trainer (and its subdirectories, including the ray-based ones), multiple trainers are implemented for different RL algorithms (PPO, DPO, KD, KTO, PRM, etc.), along with utilities such as experience makers, KL controllers, and distributed training utilities.
• Utilities: The openrlhf/utils directory includes helper functions related to logging, distributed training, deepspeed integration, and general-purpose processes.
c. External Interfaces & Deployment:
• Docker: The dockerfile directory and related shell scripts indicate containerized environments for running training/inference.
• Examples: Located in the examples folder, these scripts (for example, training on single vs. multi-node setups) act as practical usage guides and entry points.
Relationships and Interactions
• The data flow starts with the Dataset component: data preprocessing and filtering, passing validated data to the trainers.
• The Model component receives processed data, either for fine-tuning (SFT) or RL-based training. The trainers (e.g., PPO, DPO, etc.) are the orchestration layer where the actual reinforcement learning algorithms are applied using these models.
• CLI tools serve as the interface for users to trigger training, inference, or batch processing. They interact with the underlying trainers and datasets.
• Distributed training utilities (especially under the ray modules) integrate with external frameworks (e.g., vLLM) to handle scalable, multi-node training.
• Docker and CI/CD (GitHub workflows) wrap around the entire system to ensure repeatable environments and integration pipelines.
Architectural Patterns and Principles
• Modular Separation of Concerns: Components are clearly segregated into CLI, dataset processing, model definitions, and training algorithms.
• Layered Architecture: Data flows from raw input (datasets) → processing → learning (models and trainers) → external interfaces (CLI and deployment scripts).
• Distributed and Scalable Design: The separation of trainer modules (with specialized submodules for distributed training like the ray directories and vllm workers) highlights scalability through parallelism.
• Reinforcement Learning Focus: Specific design choices (e.g., omitting KL divergence, online data filtering, specialized trainers) reflect a pattern tuned towards RL for multimodal reasoning.