masworks/ml-agent

1. What this repository is

This repo is an ML-agent training and evaluation project whose released code is primarily the exploration-enriched fine-tuning stage. The codebase is built on top of LLaMA-Factory, so the architecture is centered on LLM fine-tuning, data preprocessing, inference/chat serving, and evaluation. The README explicitly says the step-wise RL stage lives in VeRL, not here.

2. Top-level execution paths

The main user entry points are:

- src/train.py for training runs

- src/webui.py for the interactive UI

- src/api.py for API serving

- scripts/train_step1.sh and scripts/eval.sh as repo-level orchestration scripts

The examples/ directory contains ready-to-run configs for SFT, LoRA, QLoRA, inference, merge, and accelerator-specific setups.

3. Configuration and experiment management

The repo is config-driven. Hyperparameters are organized under src/llamafactory/hparams/, with separate argument groups for model, data, training, generation, and evaluation. The examples/ tree shows the intended experiment matrix: full finetuning, LoRA, QLoRA, distributed training variants, and inference backends. This is the main boundary for defining an experiment before any model/data code runs.

4. Data subsystem

The data path starts in src/llamafactory/data/:

- dataset metadata and task definitions are in data/dataset_info.json

- loaders/parsers/formatters/converters normalize raw datasets into training examples

- processors split the behavior by objective: supervised, unsupervised, pretraining, pairwise, and feedback

- template.py and tool_utils.py shape prompts, chat templates, and tool-style examples

- mm_plugin.py adds multimodal handling for image/vision-language tasks

In architectural terms, this layer converts heterogeneous benchmark/task data into a unified tokenized format consumed by the trainers.

5. Model loading and adaptation

The src/llamafactory/model/ layer is responsible for instantiating base models and applying training-time modifications. It includes:

- loader.py for model construction

- adapter.py for parameter-efficient tuning paths such as LoRA-style adapters

- patcher.py and model_utils/ for runtime model edits

- support modules for quantization, RoPE, attention, checkpointing, embedding, MoE, visual components, and long-context behavior

This is the main boundary between the generic Hugging Face model stack and the project-specific training strategies.

6. Training workflows

Training is split by objective under src/llamafactory/train/:

- sft/ for supervised fine-tuning

- pt/ for pretraining

- rm/ for reward modeling

- dpo/ for preference optimization

- kto/ and ppo/ for additional alignment/RL-style methods

Each mode has a workflow.py that wires arguments, data, model, and trainer together, plus a trainer.py that encapsulates the actual optimization loop. Shared logic lives in trainer_utils.py, callbacks.py, and tuner.py.

7. Evaluation subsystem

Evaluation is handled separately from training in src/llamafactory/eval/, with an evaluator and templates for task-specific scoring. The standalone evaluation/ directory contains benchmark-specific scripts and mappings for:

- MMLU

- CMMLU

- CEval

This indicates a clean separation between model training and benchmark evaluation, with evaluation scripts adapting model outputs to benchmark scoring rules.

8. Inference, chat, and serving

The runtime inference boundary is in src/llamafactory/chat/ and src/llamafactory/api/:

- chat_model.py and engine implementations wrap either Hugging Face or vLLM

- base_engine.py, hf_engine.py, and vllm_engine.py select execution backends

- api/app.py, api/chat.py, and api/protocol.py expose serving endpoints and request/response schemas

This layer is for post-training use of the model, not for optimization. The repo also includes scripts/vllm_infer.py for direct vLLM-based inference.

9. Web UI as an orchestration layer

The src/llamafactory/webui/ package is a Gradio-style interactive front end that wraps training, inference, evaluation, data management, and export flows. Components are split by function:

- chat

- data

- eval

- infer

- train

- export

Supporting modules like runner.py, engine.py, and manager.py coordinate background execution and state management. Architecturally, the UI does not own model logic; it delegates to the same underlying training/inference subsystems.

10. Scripts and operational tooling

The repo includes utility scripts for:

- checkpoint conversion (scripts/convert_ckpt/)

- initializing specific adapter methods such as PiSSA and LoFTQ

- statistics and diagnostics (scripts/stat_utils/)

- API examples and multimodal/tool-call demos (scripts/api_example/)

These scripts are operational glue around the core training stack and help move models between formats or inspect training behavior.

11. External infrastructure and execution backends

The architecture assumes integration with the broader Hugging Face ecosystem and distributed training backends. The examples show support for:

- Accelerate

- DeepSpeed

- FSDP

- vLLM

There are also specialized QLoRA/quantization paths, which affect memory layout and backend selection. These are important architectural boundaries because they change how the same trainer executes on different hardware budgets.

12. Repository boundary relative to the ML-Agent paper

For the ML-Agent project described in the README, this repo covers the fine-tuning foundation and evaluation scaffolding, but not the full agentic training stack. The held-in/held-out ML task framing is reflected in the data/evaluation organization, while the online RL stage is intentionally externalized. If you are drawing the system diagram, treat this repository as the model adaptation and benchmark-facing half of the overall ML-Agent pipeline.