# kaistai/cot-collection architecture

> Architecture diagram of the GitHub repository kaistai/cot-collection, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-10-07). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/kaistai/cot-collection
- Repository: https://github.com/kaistai/cot-collection

## Overview

##### Project Type and Purpose
- This is a research‐oriented pipeline for Chain-of-Thought (CoT) data augmentation and fine-tuning of T5 models.
- It consists of two primary phases:
a) Rationale Augmentation – generate synthetic rationales via OpenAI APIs
b) Fine-tuning – train CoT-enhanced T5 models (3B & 11B) using DeepSpeed and Huggingface.

##### Key Components (to represent
A. Data Sources Layer
1. Original CoT Collection
– Huggingface dataset ("kaist-ai/CoT-Collection")
2. Demonstration Prompts
– Plain-text prompt files under
CoT_Rationale_Augmentation/demonstration/{task}/{prompt.txt}

B. Rationale Augmentation Module
1. Orchestration Scripts (shell)
– scripts/{flan.sh, sni.sh, t0.sh, t0p.sh, additional.sh}
2. Python Runner & Utils
– CoT_Rationale_Augmentation/run.py
– CoT_Rationale_Augmentation/utils.py
3. External Service
– OpenAI API (keys in api_keys.json → api.json)
4. Outputs
– Generated rationales saved in
CoT_Rationale_Augmentation/outputs/{subset}/{model_name}/rat/temp_{temperature}/

C. Data Extraction & Merging
1. Data Directory
– data_extraction/data/{split}/{subset}/
2. Extraction Logic
– Merges original + augmented into
codex_rationale_{split}_{phase}.json

D. Fine-Tuning Module
1. Configuration Files
– CoT_Finetuning/configs/train/*.json
– CoT_Finetuning/configs/eval/*.json
2. Training Code
– CoT_Finetuning/src/run.py
– CoT_Finetuning/src/run_peft.py
– CoT_Finetuning/src/T5.py, Datasets_end2end.py, evaluation.py, utils.py
3. DeepSpeed Engine
– CoT_Finetuning/src/deepspeed.json
4. External Libraries
– Huggingface Transformers & Datasets
5. Outputs
– Checkpoints: “kaist-ai/CoT-T5-3B” and “kaist-ai/CoT-T5-11B”

E. Model Serving / Usage
– Example in README: loading via
`transformers.AutoModelForSeq2SeqLM.from_pretrained(...)`

##### Relationships & Data Flow
1. Original Dataset + Demonstrations → (Rationale Augmentation Scripts & OpenAI API) → Augmented Rationales
2. Augmented Rationales → (Data Extraction) → Merged JSON ready for training
3. Merged JSON + Training Configs → (CoT_Finetuning with DeepSpeed & Transformers) → Fine-tuned Model
4. Fine-tuned Model → (Huggingface Hub) → End-user loading via Transformers

##### Architectural Patterns & Principles
- Pipeline/ETL structure: clear separation of augmentation, extraction, and training phases
- Modularity: each phase in its own top-level directory
- Configuration-driven: JSON configs for training & evaluation
- External dependencies managed via scripts and api_keys.json

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Data Sources Layer
    subgraph "Data Sources Layer"
        direction TB
        HF["Huggingface Dataset\nkaist-ai/CoT-Collection"]:::dataSources
        Demo["Demonstration Prompts\n{task}/prompt.txt"]:::dataSources
    end

    %% Augmentation Layer
    subgraph "Rationale Augmentation Module"
        direction TB
        Scripts_flan["flan.sh"]:::augmentation
        Scripts_sni["sni.sh"]:::augmentation
        Scripts_t0["t0.sh"]:::augmentation
        Scripts_t0p["t0p.sh"]:::augmentation
        Scripts_add["additional.sh"]:::augmentation
        Runner["run.py"]:::augmentation
        UtilsAug["utils.py"]:::augmentation
        APIKeys["api_keys.json"]:::augmentation
        OpenAICloud("OpenAI API"):::external
        AugOutputs["Generated Rationales\noutputs/{subset}/{model}/rat/temp_{temp}/"]:::augmentation
    end

    %% Extraction Layer
    subgraph "Data Extraction & Merging"
        direction TB
        RawData["Raw & Augmented Data\ndata/{split}/{subset}/"]:::extraction
        MergeLogic["Merge Logic\ncodex_rationale_{split}_{phase}.json"]:::extraction
    end

    %% Fine-Tuning Layer
    subgraph "Fine-Tuning Module"
        direction TB
        TrainCfg["Train Configs\nCoT_T5_3B.json\nCoT_T5_11B.json"]:::fineTuning
        EvalCfg["Eval Configs\nCoT_T5_cot_gen.json\nFLAN_T5_dir_cls.json"]:::fineTuning
        RunnerFT["run.py"]:::fineTuning
        RunnerPEFT["run_peft.py"]:::fineTuning
        ModelAbst["T5.py\nDatasets_end2end.py"]:::fineTuning
        EvalFT["evaluation.py\nutils.py"]:::fineTuning
        DSConfig["deepspeed.json"]:::fineTuning
        Checkpoints["Fine-Tuned Models\nCoT-T5-3B & CoT-T5-11B"]:::fineTuning
    end

    %% Serving Layer
    subgraph "Model Serving / Usage"
        direction TB
        HFHub("Huggingface Hub"):::external
        Readme["README.md"]:::external
    end

    %% Data Flow
    HF -->|"use original data"| Demo
    HF -->|"use original data"| Scripts_flan
    Demo -->|"provide prompts"| Scripts_flan
    Scripts_flan --> Runner
    Scripts_sni --> Runner
    Scripts_t0 --> Runner
    Scripts_t0p --> Runner
    Scripts_add --> Runner
    Runner --> UtilsAug
    APIKeys --> Runner
    Runner -->|"call API"| OpenAICloud
    OpenAICloud -->|"generate rationales"| AugOutputs
    AugOutputs --> RawData
    RawData --> MergeLogic
    MergeLogic --> TrainCfg
    MergeLogic --> EvalCfg
    TrainCfg --> RunnerFT
    EvalCfg --> EvalFT
    RunnerFT --> ModelAbst
    RunnerPEFT --> ModelAbst
    ModelAbst --> DSConfig
    DSConfig --> Checkpoints
    Checkpoints --> HFHub
    HFHub --> Readme

    %% Click Events
    click Demo "https://github.com/kaistai/cot-collection/blob/main/CoT_Rationale_Augmentation/demonstration/cbqa/prompt.txt"
    click Scripts_flan "https://github.com/kaistai/cot-collection/blob/main/scripts/flan.sh"
    click Scripts_sni "https://github.com/kaistai/cot-collection/blob/main/scripts/sni.sh"
    click Scripts_t0 "https://github.com/kaistai/cot-collection/blob/main/scripts/t0.sh"
    click Scripts_t0p "https://github.com/kaistai/cot-collection/blob/main/scripts/t0p.sh"
    click Scripts_add "https://github.com/kaistai/cot-collection/blob/main/scripts/additional.sh"
    click Runner "https://github.com/kaistai/cot-collection/blob/main/CoT_Rationale_Augmentation/run.py"
    click UtilsAug "https://github.com/kaistai/cot-collection/blob/main/CoT_Rationale_Augmentation/utils.py"
    click APIKeys "https://github.com/kaistai/cot-collection/blob/main/api_keys.json"
    click RawData "https://github.com/kaistai/cot-collection/blob/main/data_extraction/data/train/additional/original/additional_code_train_phase2_600.json"
    click TrainCfg "https://github.com/kaistai/cot-collection/blob/main/CoT_Finetuning/configs/train/CoT_T5_3B.json"
    click EvalCfg "https://github.com/kaistai/cot-collection/blob/main/CoT_Finetuning/configs/eval/CoT_T5_cot_gen.json"
    click RunnerFT "https://github.com/kaistai/cot-collection/blob/main/CoT_Finetuning/src/run.py"
    click RunnerPEFT "https://github.com/kaistai/cot-collection/blob/main/CoT_Finetuning/src/run_peft.py"
    click ModelAbst "https://github.com/kaistai/cot-collection/blob/main/CoT_Finetuning/src/T5.py"
    click EvalFT "https://github.com/kaistai/cot-collection/blob/main/CoT_Finetuning/src/evaluation.py"
    click EvalFT "https://github.com/kaistai/cot-collection/blob/main/CoT_Finetuning/src/utils.py"
    click DSConfig "https://github.com/kaistai/cot-collection/blob/main/CoT_Finetuning/src/deepspeed.json"
    click Readme "https://github.com/kaistai/cot-collection/blob/main/README.md"

    %% Styles
    classDef dataSources fill:#cce5ff,stroke:#333,stroke-width:1px
    classDef augmentation fill:#d4edda,stroke:#333,stroke-width:1px
    classDef extraction fill:#e2d5f9,stroke:#333,stroke-width:1px
    classDef fineTuning fill:#fff3cd,stroke:#333,stroke-width:1px
    classDef external fill:#ffe5b4,stroke:#333,stroke-width:1px
```
