# novasky-ai/skythought architecture

> Architecture diagram of the GitHub repository novasky-ai/skythought, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-01-12). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/novasky-ai/skythought
- Repository: https://github.com/novasky-ai/skythought

## Overview

##### High-Level Components

a. Data Processing Layer
- `/data` directory containing training datasets
- To the data curation tools in `skythought/tools`
- Components for:
- Data formatting
- Quality control
- Evaluation metrics
- Data preprocessing pipelines

b. Training Layer (centered around LLaMA-Factory integration)
- Training pipeline in `skythought/train`
- Integration with LLaMA-Factory components:
- Model configuration
- Training workflows
- Optimization strategies
- DeepSpeed integration

c. Evaluation Layer
- Evaluation components for:
- Math benchmarks
- Coding benchmarks
- Science benchmarks
- Performance metrics collection

##### Component Relationships

- Data flow from raw data through processing to training
- Model checkpoint handling and evaluation pipelines
- Integration points with external services (Lambda Cloud, H100 GPUs)
- Feedback loops for model improvement

##### Technical Details

a. Infrastructure Components:
- H100 GPU cluster configuration
- DeepSpeed Zero-3 offloading setup
- Batch processing systems

b. Processing Pipelines:
- Data curation workflow
- Training pipeline with 3-epoch configuration
- Evaluation benchmarking system

##### Specific Implementation Details

- Learning rate: 1e-5
- Batch size: 96
- Training duration: 19 hours
- GPU configuration: 8 H100 GPUs
- DeepSpeed Zero-3 offloading settings

##### Integration Points

Connections with:
- HuggingFace model repository
- External evaluation benchmarks
- Data sources and sinks

- Data flow through the system
- Training progression
- Evaluation feedback loops
- Model deployment paths

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Styles and classes
    classDef data fill:#b3d1ff,stroke:#333
    classDef training fill:#90EE90,stroke:#333
    classDef eval fill:#FFB366,stroke:#333
    classDef infra fill:#E0E0E0,stroke:#333
    classDef external fill:#D3D3D3,stroke:#333

    %% Data Processing Layer
    subgraph DataLayer["Data Processing Layer"]
        RawData[("Raw Data")]
        DataFormat["Data Formatting"]
        DataCombine["Data Combination"]
        DataQuality["Data Quality Control"]
        MathLabel["Math Difficulty Labeling"]
    end

    %% Training Layer
    subgraph TrainingLayer["Training Layer (LLaMA-Factory)"]
        TrainPipe["Training Pipeline"]
        DeepSpeed["DeepSpeed Integration"]
        TrainFlow["Training Workflows"]
        ModelConfig["Model Configuration"]
    end

    %% Evaluation Layer
    subgraph EvalLayer["Evaluation Layer"]
        MainEval["Main Evaluation"]
        MathTest["Math Testing"]
        CodeTest["Code Testing"]
        GenTest["General Testing"]
    end

    %% Infrastructure Components
    subgraph Infrastructure["Infrastructure"]
        DSConfig["DeepSpeed Config"]
        TrainScripts["Training Scripts"]
        MultiNode["Multi-node Support"]
        H100["H100 GPU Cluster"]
    end

    %% External Services
    ExternalSvc["External Services"]
    HuggingFace["HuggingFace Repository"]

    %% Connections
    RawData --> DataFormat
    DataFormat --> DataCombine
    DataCombine --> DataQuality
    DataQuality --> TrainPipe
    DataQuality --> MathLabel

    TrainPipe --> DeepSpeed
    ModelConfig --> TrainFlow
    TrainFlow --> DeepSpeed
    DeepSpeed --> H100

    TrainPipe --> MainEval
    MainEval --> MathTest
    MainEval --> CodeTest
    MainEval --> GenTest

    DSConfig --> DeepSpeed
    TrainScripts --> MultiNode
    MultiNode --> H100

    MainEval --> HuggingFace
    ExternalSvc --> H100

    %% Click events for component mapping
    click DataFormat "https://github.com/NovaSky-AI/SkyThought/blob/main/skythought/tools/convert_format.py"
    click DataCombine "https://github.com/NovaSky-AI/SkyThought/blob/main/skythought/tools/combine_data.py"
    click DataQuality "https://github.com/NovaSky-AI/SkyThought/blob/main/skythought/tools/inference_and_check.py"
    click MathLabel "https://github.com/NovaSky-AI/SkyThought/blob/main/skythought/tools/label_math_difficulty.py"
    click TrainPipe "https://github.com/NovaSky-AI/SkyThought/tree/main/skythought/train/LLaMA-Factory/src/llamafactory/train/"
    click DeepSpeed "https://github.com/NovaSky-AI/SkyThought/tree/main/skythought/train/LLaMA-Factory/examples/deepspeed/"
    click TrainFlow "https://github.com/NovaSky-AI/SkyThought/tree/main/skythought/train/LLaMA-Factory/src/llamafactory/train/sft/"
    click ModelConfig "https://github.com/NovaSky-AI/SkyThought/tree/main/skythought/train/LLaMA-Factory/src/llamafactory/model/"
    click MainEval "https://github.com/NovaSky-AI/SkyThought/blob/main/skythought/tools/eval.py"
    click MathTest "https://github.com/NovaSky-AI/SkyThought/blob/main/skythought/tools/util/math/testing_util.py"
    click CodeTest "https://github.com/NovaSky-AI/SkyThought/blob/main/skythought/tools/util/livecodebench/testing_util.py"
    click GenTest "https://github.com/NovaSky-AI/SkyThought/blob/main/skythought/tools/util/apps/testing_util.py"
    click DSConfig "https://github.com/NovaSky-AI/SkyThought/blob/main/skythought/train/LLaMA-Factory/zero3_config.json"
    click TrainScripts "https://github.com/NovaSky-AI/SkyThought/blob/main/skythought/train/LLaMA-Factory/train.sh"
    click MultiNode "https://github.com/NovaSky-AI/SkyThought/blob/main/skythought/train/LLaMA-Factory/launch_3_nodes.sh"

    %% Legend
    subgraph Legend
        L1[Data Components]:::data
        L2[Training Components]:::training
        L3[Evaluation Components]:::eval
        L4[Infrastructure Components]:::infra
        L5[External Services]:::external
    end
```
