How liushiliushi/conftuner works
Project Type and Purpose
• This is a machine-learning research codebase (ConfTuner) for fine-tuning large language models (LLMs) to verbalize their confidence. It provides:
• Data generation scripts that query base LLMs (Llama-3.1, Qwen, Mistral, GPT-4o) to produce question+answer+confidence data.
• A training pipeline (uncertainty_sft.py) built on HuggingFace Accelerate, DeepSpeed, and PEFT that applies a custom Tokenized Brier Score loss.
• Inference and evaluation scripts (inference.py, inference_gpt.py) to measure accuracy and calibration, optionally logging to Weights & Biases.
• A rich configuration hierarchy under src/llama_recipes (datasets, inference, training, quantization, etc.).
File Structure Analysis
• Top-level
• generate_response.py, inference.py, inference_gpt.py, uncertainty_sft.py: entry-point scripts
• scripts.sh, train_inference.sh: helper shell wrappers
• downloaded_config/adapter_config.json: adapter/hyperparameter defaults
• images/: documentation figures
• src/llama_recipes/
• configs/: JSON/YAML configs for DeepSpeed, PEFT, training, quantization, datasets, wandb
• data/: dataset loaders, samplers, formatters (including llama_guard submodule)
• inference/: model loading, prompt formatting, safety checks
• model_checkpointing/: checkpoint save/load handlers
• policies/: precision, optimizer, checkpoint-wrapping rules
• tools/: weight conversion and comparison utilities
• trained_calibration/: RL-based calibration scripts (DPO, reward models, evaluation)
• utils/: metrics computation, config parsing, FLOP counting, etc.
Key README Insights
• Two-stage method: (1) extract probability distribution over confidence tokens, (2) fine-tune with Tokenized Brier Score
• Dependencies: HuggingFace Accelerate, DeepSpeed, PEFT (LoRA/QLoRA), OpenAI API for GPT-4o evaluation, WandB
• Example commands for data generation, fine-tuning on different base models, and inference
a. Main Components
• Data Generation Module
• Training Module
• Inference/Evaluation Module
• Configuration Store
• Model Checkpoint Storage
• External Services: OpenAI API, Weights & Biases
• Compute Infrastructure: GPU cluster via Accelerate + DeepSpeed
b. Interactions and Data Flow
1. Data Generation
– generate_response.py reads Config → queries LLM (via HuggingFace or OpenAI API) → produces JSONL datasets
2. Training
– uncertainty_sft.py loads dataset + configs (DeepSpeed, PEFT, training hyperparams) → launches Accelerate/DeepSpeed on GPUs → writes checkpoints to storage → logs metrics to WandB
3. Inference/Evaluation
– inference.py / inference_gpt.py load checkpoint + dataset → run batched evaluation through LLM → calculate calibration metrics → log to WandB or stdout
c. Architectural Patterns
• Modular recipe-based design (llama_recipes) to separate concerns: data, configs, inference, policies, utils
• Configuration-driven pipeline: JSON/YAML files control DeepSpeed, PEFT, dataset sampling
• Wrapper scripts (shell and Python) for reproducible commands
d. Technologies & Frameworks
• Python 3.x, HuggingFace Accelerate, DeepSpeed, PEFT (LoRA/QLoRA)
• OpenAI SDK for GPT-4o
• Weights & Biases for experiment tracking
• JSON/YAML for configurations
• Shell scripts for orchestration
Summary
• Break the system into Data Generation, Training, Inference, Config Management, Storage, and External Services.
• How JSON/YAML configs drive each stage.
• Compute orchestration via Accelerate/DeepSpeed and experiment tracking via WandB.