# kanishkg/cognitive-behaviors architecture

> Architecture diagram of the GitHub repository kanishkg/cognitive-behaviors, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/kanishkg/cognitive-behaviors
- Repository: https://github.com/kanishkg/cognitive-behaviors

## Overview

##### Project Type and Purpose
- This project is a research and training framework for “Self-Improving Reasoners” built on top of TinyZero/DeepSeek R1 and the VERL ecosystem. It combines data generation (including countdown datasets and chain-of-thought (CoT) priming data), model pretraining (with synthetic data, SFT, and PPO training pipelines), and robust evaluation (behavioral evaluations and extensive tests).
- The repository includes multiple pipelines: data preparation, priming data generation, training (SFT and PPO), pretraining analysis, and evaluation. It also implements distributed training and inference utilities (with directories like “verl/workers” and “verl/single_controller”) along with support for multi-model configurations and even Docker deployment.

##### Analyzing the File Structure
- Several top-level directories indicate specialized components:
▪ “behavioral_evals”: Contains scripts to run behavioral evaluations (e.g., API evaluations, GPT API evaluations, and data generation for behaviors).
▪ “examples”: Provides various shell scripts and notebooks for different tasks (data preprocessing, generation, trainers for SFT/PPO, Ray-based tutorials, etc.).
▪ “generate_cot_datasets”: Holds components for generating datasets (CoT data, empty CoT, and filtering out all-positive examples).
▪ “pretraining_analysis”: Contains scripts for relabeling, computing statistics, and formatting pretraining datasets.
▪ “scripts”: Bundles commands to invoke key processes like behavioral evals, training scripts, SFT training, etc.
▪ “tests”: Includes integration, end-to-end tests, GPU utilities, and sanity checks to ensure system correctness.
▪ “verl”: The core directory that encapsulates the models (e.g., “verl/models”), trainers (e.g., “verl/trainer”), distributed controllers (e.g., “verl/single_controller”), workers, utilities (e.g., logging, distributed, debugging), and third-party integrations (e.g., vllm versions).
- Also, configuration and docker files indicate that both local and container-based setups are supported.

##### Core Architectural Components
a. Data Generation and Preprocessing Layer:
- Data preprocessing (in “examples/data_preprocess” and “tasks”) for creating raw datasets (e.g., countdown data).
- Priming data generation pipelines (under “generate_cot_datasets” and “pretraining_analysis”) that call external APIs (such as with claude-3.5-sonnet) and process outputs.
- Conversion scripts (e.g., parquet conversion) in “scripts” and “pretraining_analysis”.
b. Training Modules:
- SFT (Supervised Fine-Tuning) and PPO (Proximal Policy Optimization) pipelines run via scripts inside “scripts/” and training control in “verl/trainer”.
- Configuration files (YAMLs in “verl/trainer/config” and other config files) direct training parameters.
- Integration of multi-GPU/multi-node support (via “docker” and distributed worker scripts in “verl/workers” and “verl/single_controller”).
c. Model and Worker Management:
- The “verl” directory has model loading, checkpoint management (e.g., “verl/models/llama/megatron/checkpoint_utils”), and both Megatron and vllm engine integrations.
- Distributed training and inference workers are handled through “verl/single_controller” and “verl/workers”. These components integrate with Ray for orchestration, as evidenced by scripts and tests (e.g., “tests/ray”).
d. Evaluation and Testing:
- Behavioral evaluations (under “behavioral_evals” and “scripts/behavioral_evals.sh”) feed data to benchmark model performance.
- A comprehensive set of tests under “tests” (e2e, GPU utilities, rollout, and sanity tests) ensure correctness and performance.
e. Deployment and Utilities:
- Dockerfiles in the “docker” folder and documentation in “docs” provide instructions for containerized deployments.
- Utility modules (logger, distributed helpers, benchmarking utilities) across “verl/utils”.

##### Relationships and Data Flow
- From preprocessing, data moves into the priming data generation pipelines (using generate_cot_datasets and pretraining_analysis) to create the CoT and QA datasets.
- The prepared datasets then feed into the Training Pipeline where a base model is fine-tuned via SFT and subsequently via reinforcement learning (PPO).
- The resulting trained models flow into the evaluation subsystems — behavioral evaluations (using API evaluators and scripts) — and are continually monitored via tests.
- Distributed orchestration using Ray and internal controllers (under “verl/single_controller”) manages the parallel execution across multiple GPUs/nodes.
- Utility libraries support logging, checkpointing, and flow control throughout.

## Diagram (Mermaid)

```mermaid
graph TD
    %% Data Generation & Preprocessing Module
    subgraph "Data Generation & Preprocessing Module"
        DP1["Examples Data Preprocess"]:::dp
        DP2["Countdown Script"]:::dp
    end

    %% Priming Data Generation Module
    subgraph "Priming Data Generation Module"
        PG1["CoT Generation Scripts"]:::pg
        PG2["Pretraining Analysis Scripts"]:::pg
    end

    %% Training Module (SFT & PPO)
    subgraph "Training Module (SFT & PPO)"
        TS1["SFT Training Script"]:::training
        TS2["Train Script"]:::training
        TS3["Trainer Codebase (SFT & PPO)"]:::training
    end

    %% Model & Worker Management Module
    subgraph "Model & Worker Management Module"
        MW1["Model Management & Checkpointing"]:::mw
        MW2["Distributed Controller"]:::mw
        MW3["Worker Implementations"]:::mw
    end

    %% Evaluation & Testing Module
    subgraph "Evaluation & Testing Module"
        ET1["Behavioral Evals"]:::et
        ET2["Testing Suites"]:::et
    end

    %% Deployment & Utilities Module
    subgraph "Deployment & Utilities Module"
        DU1["Docker Deployment"]:::du
        DU2["Documentation"]:::du
        DU3["Utility Libraries"]:::du
    end

    %% Data Flow Connections
    DP1 -->|"generates"| PG1
    DP2 -->|"generates"| PG1

    PG1 -->|"feeds"| TS1
    PG2 -->|"feeds"| TS1

    TS3 -->|"triggers"| MW1
    TS3 -->|"triggers"| MW2
    TS3 -->|"triggers"| MW3

    TS3 -->|"evaluates"| ET1
    ET2 -->|"feedback"| TS3

    %% Deployment & Utilities support
    DU3 -->|"supports"| TS3
    DU3 -->|"supports"| MW1
    DU1 -->|"deploys"| MW2

    %% Click Events
    click DP1 "https://github.com/kanishkg/cognitive-behaviors/tree/main/examples/data_preprocess"
    click DP2 "https://github.com/kanishkg/cognitive-behaviors/blob/main/tasks/countdown.py"
    click PG1 "https://github.com/kanishkg/cognitive-behaviors/tree/main/generate_cot_datasets"
    click PG2 "https://github.com/kanishkg/cognitive-behaviors/tree/main/pretraining_analysis"
    click TS1 "https://github.com/kanishkg/cognitive-behaviors/blob/main/scripts/sft.sh"
    click TS2 "https://github.com/kanishkg/cognitive-behaviors/blob/main/scripts/train.sh"
    click TS3 "https://github.com/kanishkg/cognitive-behaviors/tree/main/verl/trainer"
    click MW1 "https://github.com/kanishkg/cognitive-behaviors/tree/main/verl/models"
    click MW2 "https://github.com/kanishkg/cognitive-behaviors/tree/main/verl/single_controller"
    click MW3 "https://github.com/kanishkg/cognitive-behaviors/tree/main/verl/workers"
    click ET1 "https://github.com/kanishkg/cognitive-behaviors/tree/main/behavioral_evals"
    click ET2 "https://github.com/kanishkg/cognitive-behaviors/tree/main/tests"
    click DU1 "https://github.com/kanishkg/cognitive-behaviors/tree/main/docker"
    click DU2 "https://github.com/kanishkg/cognitive-behaviors/tree/main/docs"
    click DU3 "https://github.com/kanishkg/cognitive-behaviors/tree/main/verl/utils"

    %% Styles
    classDef dp fill:#9fcdff,stroke:#333,stroke-width:2px;
    classDef pg fill:#a0e7e5,stroke:#333,stroke-width:2px;
    classDef training fill:#a8e6cf,stroke:#333,stroke-width:2px;
    classDef mw fill:#ffd3b6,stroke:#333,stroke-width:2px;
    classDef et fill:#ffaaa5,stroke:#333,stroke-width:2px;
    classDef du fill:#d291bc,stroke:#333,stroke-width:2px;
```
