How mst272/llm-dojo works
• Data & Configuration:
• Data Source: The “data” directory hosts JSONL files that supply SFT, RLHF, and potentially VLM training data.
• Configuration Files: There are Deepspeed configuration files (in “train_args/deepspeed_config” and “rlhf/ds_config”) and various argument scripts to configure the training (e.g., “train_args/common_args.py”, “rlhf/rlhf_args/…”).
• SFT Training Module:
• Core Training Script: “main_train.py” is the entry point for standard training.
• Training Arguments: “train_args” directory provides SFT-specific configurations, including scripts to adapt to chat templates and support changes (Lora, Qlora, etc.).
• Data Processing: “utils/script/generate_data.py” plays a role in adapting raw instruction-output files into the required format.
• RLHF Training Module:
• RLHF Framework: The “rlhf” directory encompasses the RLHF training pipeline, including methods like PPO, DPO, RLOO, and SimPO.
• Knowledge Distillation & Rejected Sampling: Subsections within “rlhf” such as “rejected_sampling” and the “train_rlhf.py” / “train_gkd.py” scripts illustrate the handling of various RLHF techniques.
• Argument Management: Similar to SFT, RLHF has its own set of configuration and argument files under “rlhf/rlhf_args”.
• Evaluation Module:
• Located under the “evaluate” directory, it contains scripts for setting up and running evaluations (e.g., “evaluation.py”, “execution.py”, and “dataset” with Humaneval examples).
• Task-specific Scripts: The “evaluate/task” and “evaluate/generate.py” modules inform how model outputs are tested.
• VLM (Visual Language Modeling) Module:
• Inference and Training: “inference_vllm.py” and “vlm_train.py” along with a “train_vlm_config” directory (under “train_args/vlm_config”) provide support for VLM tasks.
• Multi-modal Components: The README outlines VLM support (QA tasks using vision and text) with configuration options for freezing visual components.
• LLM Tricks & Experimental Implementations:
• The “llm_tricks” directory holds examples and experimental code for various advanced techniques. Each subfolder (e.g., “DPO_example”, “dora”, “moe”, “transformer”) represents independent experimental modules offering innovative training/trick approaches.
• Utility & Helper Components:
• “utils” contains helper functions for data collation, processing, evaluation utilities, and scripts for tasks like downloading models or merging LoRA files.
Relationships and Interactions
• That both the SFT and RLHF modules generate trained models that further feed into the Evaluation Module.
• That “llm_tricks” serves as a sandbox for experiments, which may also depend on similar data and configuration inputs.
Architectural Patterns and Principles
• Modularity: The clear separation between “sft”, “rlhf”, “vlm”, “evaluate”, and “llm_tricks” illustrates a modular architecture where each major training method or experiment encapsulates its logic.
• Configuration-Driven: Notice the use of parameter files and config scripts (Deepspeed YAML/JSON and common_args Python modules) that allow dynamic adjustments without deep code changes.
• Reusability: The use of “utils” and argument management directories across both SFT and RLHF promotes code reuse across modules.
• Extensibility: The “llm_tricks” submodules highlight an extensible design – each experimental method is isolated and independently documented.