# ruckbreasoning/omnisql architecture

> Architecture diagram of the GitHub repository ruckbreasoning/omnisql, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/ruckbreasoning/omnisql
- Repository: https://github.com/ruckbreasoning/omnisql

## Overview

##### Project Type and Purpose
- This is a full-fledged research and engineering project that builds a scalable text-to-SQL data synthesis framework. The project synthesizes high-quality text-to-SQL data (SynSQL-2.5M) using multiple pipelines (chain-of-thought, database, question, and SQL synthesis), and then leverages these datasets to train and evaluate models (OmniSQL-7B/14B/32B).
- The README emphasizes both the data synthesis framework and the training/evaluation pipeline, indicating that the system encompasses both data generation and model fine-tuning/inference.

##### Analyzing the File Structure
- The top-level directories include:
- data_synthesis – Contains submodules for each synthesis task:
▪ cot_synthesis – Responsible for generating chain-of-thought solutions.
▪ database_synthesis – Handles schema enhancement, database creation (SQLite focused), and prompt generation.
▪ question_synthesis – Focuses on generating and processing natural language questions.
▪ sql_synthesis – Generates SQL queries and handles their post-processing.
- train_and_evaluate – Houses scripts and configuration files for training models (with various model sizes) and running evaluation scripts (including integration with datasets like Spider, BIRD, etc.).
- examples – Provides sample prompts used in tests/inference.
- Notice the modular arrangement; each synthesis component has its own README, prompt templates, and processing scripts. The train_and_evaluate folder includes configuration (e.g., accelerate YAML files) and multiple scripts for each training and evaluation step.

##### Final Steps
- Then, expand each major component (with data_synthesis modules and the train_and_evaluate modules) to show internal details.

## Diagram (Mermaid)

```mermaid
graph TD
    %% Data Synthesis Framework
    subgraph "Data Synthesis Framework"
        DS["Data Synthesis Pipeline"]:::synthesis
        CS["Cot Synthesis Module"]:::synthesis
        DB["Database Synthesis Module"]:::synthesis
        QS["Question Synthesis Module"]:::synthesis
        SS["SQL Synthesis Module"]:::synthesis
        Dataset["SynSQL-2.5M Dataset"]:::dataset

        CS -->|"feeds"| Dataset
        DB -->|"feeds"| Dataset
        QS -->|"feeds"| Dataset
        SS -->|"feeds"| Dataset
    end

    %% Training and Evaluation Pipeline
    subgraph "Training and Evaluation Pipeline"
        TE["Training & Evaluation Pipeline"]:::training
        Eval["Model Evaluation"]:::training
        Dataset -->|"used_in"| TE
        TE -->|"triggers"| Eval
    end

    %% Inference/Integration Component
    Infer["Inference/Integration Component"]:::inference
    TE -->|"deploys"| Infer

    %% Visual Assets for Documentation
    Assets["Visual Assets"]:::asset
    Infer -->|"documents"| Assets

    %% Styles
    classDef synthesis fill:#FEE,stroke:#333,stroke-width:2px;
    classDef dataset fill:#FF9,stroke:#333,stroke-width:2px;
    classDef training fill:#CFF,stroke:#333,stroke-width:2px;
    classDef inference fill:#CFC,stroke:#333,stroke-width:2px;
    classDef asset fill:#FC9,stroke:#333,stroke-width:2px;

    %% Click Events
    click DS "https://github.com/ruckbreasoning/omnisql/tree/main/data_synthesis"
    click CS "https://github.com/ruckbreasoning/omnisql/tree/main/data_synthesis/cot_synthesis"
    click DB "https://github.com/ruckbreasoning/omnisql/tree/main/data_synthesis/database_synthesis"
    click QS "https://github.com/ruckbreasoning/omnisql/tree/main/data_synthesis/question_synthesis"
    click SS "https://github.com/ruckbreasoning/omnisql/tree/main/data_synthesis/sql_synthesis"
    click TE "https://github.com/ruckbreasoning/omnisql/tree/main/train_and_evaluate"
    click Infer "https://github.com/ruckbreasoning/omnisql/blob/main/train_and_evaluate/infer.py"
    click Assets "https://github.com/ruckbreasoning/omnisql/tree/main/assets"
```
