How rucaibox/llmbox works
Project Type and Purpose
• LLMBox is a modular machine‐learning library for (a) training large language models (LLMs) via diverse strategies (SFT, PT, PPO, DPO, LoRA, QLoRA) and (b) utilizing/evaluating models across 59+ datasets with various evaluation methods (get_ppl, get_prob, generation) and acceleration backends (vLLM, Flash Attention, Deepspeed).
• This is neither a web app nor a monolithic service but a CLI‐driven data‐processing & model‐training/evaluation framework.
File Structure (High‐Level)
• Top‐level “inference.py”: single‐entry CLI for utilization.
• Top‐level “train.py”: single‐entry CLI for training.
• Directories:
• training/: data processing, dataset modules, strategy scripts, configs, utilities for training.
• utilization/: dataset loaders, model wrappers, evaluator, metrics, utilities for inference/evaluation.
• docs/, tests/: documentation and test harness.
• Config files: JSON deepspeed configs, requirements, bash scripts for different training presets.
• External‐service integrations: Huggingface Transformers, OpenAI API, Anthropic, vLLM, Flash Attention, Deepspeed.
README for Insights
• Confirms two main pipelines: Training and Utilization.
• Lists supported strategies, datasets, performance benchmarks.
Main Components & Interactions
a. CLI Layer
• train.py → invokes Training Pipeline
• inference.py → invokes Utilization Pipeline
• Parses arguments via utilization/utils/arguments.py (shared parser modules).
b. Training Pipeline
• Config Loader: JSON configs (training/configs/) and CLI args
• Dataset Builder:
• training/dataset/pt_dataset.py (Pretrain)
• training/dataset/sft_dataset/… (SFT datasets)
• Tokenizer Merger: training/merge_tokenizer.py
• Training Strategies:
• SFT (training/train.py + training/ppo.py + training/dpo.py)
• PT (training/pt.py)
• Parameter‐Efficient (LoRA, QLoRA) via Deepspeed
• Compute Engines: Huggingface Trainer / Flash‐Attention / Deepspeed
• Artifacts: checkpoint & model outputs, logs.
c. Utilization Pipeline
• Config & Argument Parser (shared with CLI)
• Model Loader:
• utilization/model/*.py (Huggingface, OpenAI, Anthropic, vLLM)
• Dataset Loader:
• utilization/dataset/*.py + helper mixins
• Supports 59+ datasets via dataset_utils
• Prompt & ICL/COT Module: utilization/chat_templates.py, dataset_utils/icl_strategies.py
• Inference Engine: Transformers vs. vLLM vs. OpenAI API, with prefix caching
• Evaluator & Metrics: utilization/evaluator.py, utilization/metric/*
• Result Logger: utilization/utils/log_results.py
d. External Services & Dependencies
• Huggingface Hub (model & tokenizer fetch)
• OpenAI API (for chat‐style models)
• vLLM library
• Deepspeed & Flash Attention
• BitsAndBytes / GPTQ quantization backends
Architectural Patterns & Principles
• Modular Pipeline Pattern: clear separation of dataset, model, training/inference strategy, evaluation.
• CLI‐driven Orchestration: single entrypoints dispatch to submodules.
• Dependency Injection: pass configs/strategies via JSON or CLI.
• Strategy Pattern: interchangeable training strategies (SFT, PT, PPO, DPO) and evaluation methods (get_ppl, get_prob, generation).
• Adapter Pattern: model wrappers for different backends (Huggingface, OpenAI, Anthropic).