How contextualai/halos works
Project type and purpose
• This is an Open-Source, modular ML training framework (“HALOs”) for aligning large language models (LLMs) with various human‐centered loss functions (DPO, KTO, PPO, etc.).
• It provides:
• CLI entry points (launch.py, label.py)
• Hydra‐based config management (config/, accelerate_config/)
• Data loading, training, sampling, and logging modules
• Multi‐GPU/FSDP orchestration via Accelerate
• Easy extensibility for new losses, dataloaders, trainers, and models
File structure
– Top‐level:
• launch.py & label.py: main CLI entrypoints
• install.sh, scripts/: environment setup and multi‐node launch scripts
• accelerate_config/: ready‐to‐use Hugging Face Accelerate FSDP configs
• config/: Hydra configs for losses, models, global settings
• examples/: sample feedback data formats
• train/: core library modules (dataloader, models, trainers, sampling, utils)
– Patterns & architectural choices:
• Plugin pattern: new loss = yaml + trainer subclass + dataloader
• Hydra for hierarchical config composition
• Accelerate + FSDP for distributed training
• Separation of concerns: CLI, config, data, training, evaluation
README for additional insights
• Core stacks: Python, PyTorch, Hydra, Accelerate, FSDP, Wandb (for logging), Hugging Face Hub
• Execution flow:
1. User runs accelerate launch ... launch.py loss=X model=Y datasets=[...] ...
2. Hydra instantiates the config hierarchy (loss, model, trainer settings)
3. Modules in train/ are wired together: dataloader → model → trainer → sampler/label
4. Accelerate wraps the training loop to spawn processes, handle inter‐GPU comms via NCCL
5. Checkpoints & metrics saved to disk and logged to Wandb
a. Main components
• User CLI: launch.py, label.py
• Config Layer: Hydra (config/*.yaml), Accelerate (accelerate_config/*.yaml)
• Data Layer: Dataloaders in train/dataloader.py + examples/
• Model Layer: train/models.py (LLM wrappers, PEFT/LoRA)
• Training Layer: train/trainers.py, train/sample.py (trainer subclasses, sampling logic)
• Orchestration: Accelerate & FSDP (multi‐GPU, multi‐node management)
• Logging & Checkpointing: Wandb, local disk (cache_dir)
• External Services: Hugging Face Hub (model & dataset), Wandb, openAI API (for evaluation)
b. Relationships and interactions
1. User issues CLI command → invokes launch.py
2. launch.py uses Hydra to parse & merge configs (global + loss + model + datasets)
3. Hydra config drives instantiation of:
– DataLoader (e.g., UnpairedPreferenceDataLoader)
– Trainer (e.g., KTOTrainer, DPOTrainer)
– Model (LLaMA, Mistral, Gemma, etc.)
4. Accelerate intercepts the Python entrypoint to:
– Spawn N processes across GPUs/nodes
– Wrap the Trainer’s train() loop in FSDP for model parallelism
– Handle distributed backend (NCCL) sync & checkpoint broadcasting
5. Trainer calls DataLoader → yields batches → forward pass through Model
6. Loss computed by Trainer → backward pass → FSDP optimizer step
7. Metrics & checkpoints logged to Wandb & disk
8. Optional: label.py uses trained model to generate feedback datasets
c. Architectural patterns & design principles
– Modular plugin architecture: new losses/models/dataloaders as independent units
– Separation of concerns: config vs code vs orchestration vs data
– Declarative config (Hydra) driving imperative code
– Distributed training abstraction (Accelerate + FSDP)
– Extensibility: add yaml + subclass to plug into existing pipeline
d. Key technologies & frameworks
– Python 3, PyTorch
– Hydra for config management
– Hugging Face Accelerate for distributed training & FSDP
– Wandb for experiment tracking
– Hugging Face Hub for model/dataset I/O
– Optional: FlashAttention, PEFT/LoRA
Final advice
– Be as detailed as possible: show every plugin boundary and integration point
– Don’t over‐compress: capture all Hydra config layers (global, loss, model)
– Show both control flow (CLI → Trainer) and data flow (batches → model → gradients)