How foundation-model-stack/fms-dgt works
Project Type and Core Purpose
This is a framework for generating synthetic data using Large Language Models (LLMs). The system is designed to be modular, scalable, and supports multiple LLM providers.
Main Components
a. Core Framework Components:
• Base Layer (fms_dgt/base/): Contains fundamental abstractions
• Blocks Layer (fms_dgt/blocks/): Implements core functionalities
• DataBuilders Layer (fms_dgt/databuilders/): Handles data generation logic
• Resources Layer (fms_dgt/resources/): Manages external resources
b. External Integrations:
• LLM Providers:
• IBM GenAI
• OpenAI
• vLLM
• RITS
• Data Processing Tools:
• IBM Data Prep Kit (DPK)
┌─ External Layer ─────────────────────┐
• [LLM Providers] [Processing Tools] │
• ┘
• ↑
┌─ Resource Management Layer ──────────┐
• [API Handlers] [GPU Management] │
• ┘
• ↑
┌─ Core Processing Layer ──────────────┐
• [Generators] [Validators] [Pipeline] │
• ┘
• ↑
┌─ Data Management Layer ─────────────┐
• [DataBuilders] [DataLoaders] [Store]│
• ┘
Specific Implementation Details
┌─ Blocks ─────────┐
• Generators │
• Validators │
• Postprocessors │
• Trainers │
• ┘
Additional Elements
• A configuration section showing the relationship between .env files and API configurations
• Parallel processing capabilities
• Validation and quality control points