How kanishkg/cognitive-behaviors works
Project Type and Purpose
• This project is a research and training framework for “Self-Improving Reasoners” built on top of TinyZero/DeepSeek R1 and the VERL ecosystem. It combines data generation (including countdown datasets and chain-of-thought (CoT) priming data), model pretraining (with synthetic data, SFT, and PPO training pipelines), and robust evaluation (behavioral evaluations and extensive tests).
• The repository includes multiple pipelines: data preparation, priming data generation, training (SFT and PPO), pretraining analysis, and evaluation. It also implements distributed training and inference utilities (with directories like “verl/workers” and “verl/single_controller”) along with support for multi-model configurations and even Docker deployment.
Analyzing the File Structure
• Several top-level directories indicate specialized components:
▪ “behavioral_evals”: Contains scripts to run behavioral evaluations (e.g., API evaluations, GPT API evaluations, and data generation for behaviors).
▪ “examples”: Provides various shell scripts and notebooks for different tasks (data preprocessing, generation, trainers for SFT/PPO, Ray-based tutorials, etc.).
▪ “generate_cot_datasets”: Holds components for generating datasets (CoT data, empty CoT, and filtering out all-positive examples).
▪ “pretraining_analysis”: Contains scripts for relabeling, computing statistics, and formatting pretraining datasets.
▪ “scripts”: Bundles commands to invoke key processes like behavioral evals, training scripts, SFT training, etc.
▪ “tests”: Includes integration, end-to-end tests, GPU utilities, and sanity checks to ensure system correctness.
▪ “verl”: The core directory that encapsulates the models (e.g., “verl/models”), trainers (e.g., “verl/trainer”), distributed controllers (e.g., “verl/single_controller”), workers, utilities (e.g., logging, distributed, debugging), and third-party integrations (e.g., vllm versions).
• Also, configuration and docker files indicate that both local and container-based setups are supported.
Core Architectural Components
a. Data Generation and Preprocessing Layer:
• Data preprocessing (in “examples/data_preprocess” and “tasks”) for creating raw datasets (e.g., countdown data).
• Priming data generation pipelines (under “generate_cot_datasets” and “pretraining_analysis”) that call external APIs (such as with claude-3.5-sonnet) and process outputs.
• Conversion scripts (e.g., parquet conversion) in “scripts” and “pretraining_analysis”.
b. Training Modules:
• SFT (Supervised Fine-Tuning) and PPO (Proximal Policy Optimization) pipelines run via scripts inside “scripts/” and training control in “verl/trainer”.
• Configuration files (YAMLs in “verl/trainer/config” and other config files) direct training parameters.
• Integration of multi-GPU/multi-node support (via “docker” and distributed worker scripts in “verl/workers” and “verl/single_controller”).
c. Model and Worker Management:
• The “verl” directory has model loading, checkpoint management (e.g., “verl/models/llama/megatron/checkpoint_utils”), and both Megatron and vllm engine integrations.
• Distributed training and inference workers are handled through “verl/single_controller” and “verl/workers”. These components integrate with Ray for orchestration, as evidenced by scripts and tests (e.g., “tests/ray”).
d. Evaluation and Testing:
• Behavioral evaluations (under “behavioral_evals” and “scripts/behavioral_evals.sh”) feed data to benchmark model performance.
• A comprehensive set of tests under “tests” (e2e, GPU utilities, rollout, and sanity tests) ensure correctness and performance.
e. Deployment and Utilities:
• Dockerfiles in the “docker” folder and documentation in “docs” provide instructions for containerized deployments.
• Utility modules (logger, distributed helpers, benchmarking utilities) across “verl/utils”.
Relationships and Data Flow
• From preprocessing, data moves into the priming data generation pipelines (using generate_cot_datasets and pretraining_analysis) to create the CoT and QA datasets.
• The prepared datasets then feed into the Training Pipeline where a base model is fine-tuned via SFT and subsequently via reinforcement learning (PPO).
• The resulting trained models flow into the evaluation subsystems — behavioral evaluations (using API evaluators and scripts) — and are continually monitored via tests.
• Distributed orchestration using Ray and internal controllers (under “verl/single_controller”) manages the parallel execution across multiple GPUs/nodes.
• Utility libraries support logging, checkpointing, and flow control throughout.