How novasky-ai/skythought works
High-Level Components
a. Data Processing Layer
• /data directory containing training datasets
• To the data curation tools in skythought/tools
• Components for:
• Data formatting
• Quality control
• Evaluation metrics
• Data preprocessing pipelines
b. Training Layer (centered around LLaMA-Factory integration)
• Training pipeline in skythought/train
• Integration with LLaMA-Factory components:
• Model configuration
• Training workflows
• Optimization strategies
• DeepSpeed integration
c. Evaluation Layer
• Evaluation components for:
• Math benchmarks
• Coding benchmarks
• Science benchmarks
• Performance metrics collection
Component Relationships
• Data flow from raw data through processing to training
• Model checkpoint handling and evaluation pipelines
• Integration points with external services (Lambda Cloud, H100 GPUs)
• Feedback loops for model improvement
Technical Details
a. Infrastructure Components:
• H100 GPU cluster configuration
• DeepSpeed Zero-3 offloading setup
• Batch processing systems
b. Processing Pipelines:
• Data curation workflow
• Training pipeline with 3-epoch configuration
• Evaluation benchmarking system
Specific Implementation Details
• Learning rate: 1e-5
• Batch size: 96
• Training duration: 19 hours
• GPU configuration: 8 H100 GPUs
• DeepSpeed Zero-3 offloading settings
Integration Points
Connections with:
• HuggingFace model repository
• External evaluation benchmarks
• Data sources and sinks
• Data flow through the system
• Training progression
• Evaluation feedback loops
• Model deployment paths