# eric-ai-lab/soft-thinking architecture

> Architecture diagram of the GitHub repository eric-ai-lab/soft-thinking, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-02). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/eric-ai-lab/soft-thinking
- Repository: https://github.com/eric-ai-lab/soft-thinking

## Overview

##### Project Type and Purpose
- This is a research‐oriented LLM inference and evaluation framework (“Soft Thinking”) implementing the techniques from the paper “Soft Thinking: Unlocking the Reasoning Potential of LLMs….”
- It orchestrates data loading, model download, baseline and “soft thinking” inference modes, optional LLM‐based judgments, and result upload to Hugging Face.

##### Top‐Level Architecture
- CLI / Orchestration Layer
- Root scripts (`run_sglang_softthinking.py`, `run_sglang_nothinking.py`, `demo.py`, `codeeval.py`, `humanevaleval.py`, etc.) and shell wrappers in `scripts/baseline` and `scripts/st`.
- Responsibilities: parse args, set up environment, invoke core Python package.

- Data Layer
- `datasets/`: JSON and helper‐py modules load benchmarks (HumanEval, MBPP, AIME, etc.).
- `models/download.py`: fetches model artifacts into `models/`.

- Core Inference Package
- `sglang_soft_thinking_pkg/python/sglang`: the main Python API and orchestration logic.
- Subcomponents:
· `lang/`: compiler, interpreter, IR, chat templates, backends for OpenAI, Anthropic, VertexAI, LiteLLM, runtime endpoint.
· `srt/`: high‐performance inference engine, dispatching to CPU/GPU/triton/ROCm kernels, quantization, MoE, speculative sampling, multi‐GPU distributed.
· `launch_server.py` / `launch_router.py`: internal HTTP endpoints or router components.
· `models/`, `connector/`, `managers/`, `model_loader/`: configuration, data parallel/distributed state, memory/cache management.

- Kernel Module
- `sgl-kernel`: custom C++/CUDA/HIP kernel implementations for attention, GEMM, quantization, MoE, speculative sampling.
- Python bindings (`sgl_kernel/python/sgl_kernel`) expose ops to the SRT engine.

- Router Service
- `sgl-router`: Rust‐based routing service for distributing inference requests or coordinating multi‐host setups.

- External Services
- LLM Judge via API (OpenAI/Anthropic/Vertex) for evaluating sample quality.
- Hugging Face (optional) for pushing results.

##### Relationships & Data Flows
1. User runs a shell script or Python entrypoint.
2. Orchestration layer loads dataset, downloads or locates model.
3. Calls into `sglang` API, which compiles prompts, sets sampling parameters (soft thinking vs baseline).
4. `sglang.srt` engine dispatches to the `sgl-kernel` library (via Python bindings) to execute forward passes, attention, GEMM, MoE, quantization on GPU/CPU.
5. Partial outputs may be sent back to the interpreter for “after thinking” passes.
6. Completed samples can optionally be forwarded to an external LLM judge via HTTP.
7. Final results are collected, evaluated (e.g., correctness metrics), and optionally pushed to Hugging Face.

##### Architectural Patterns and Principles
- Modular layering: CLI → Python API → Inference engine → Native kernels.
- Plug‐in backends: abstract base classes in `lang/backend`, concrete connectors for OpenAI, Anthropic, etc.
- Separation of concerns: data loading, model loading, prompt compilation, sampling strategy, kernel execution, result evaluation.
- High-performance: custom kernels (CUDA/HIP), Triton integration, quantization schemes, caching, distributed communication.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% CLI / Orchestration Layer
    subgraph "CLI / Orchestration" 
        direction TB
        CLI1["run_sglang_softthinking.py"]:::cli
        CLI2["run_sglang_nothinking.py"]:::cli
        CLI3["demo.py"]:::cli
        CLI4["codeeval.py"]:::cli
        CLI5["humanevaleval.py"]:::cli
        CLI6["matheval.py"]:::cli
        CLI7["mbppeval.py"]:::cli
        ShellBase["scripts/baseline/"]:::cli
        ShellST["scripts/st/"]:::cli
    end

    %% Data Layer
    subgraph "Data Layer"
        direction TB
        DataLoad["datasets/"]:::python
        ModelDL["models/download.py"]:::python
    end

    %% Core Inference Package
    subgraph "Python API: sglang" 
        direction TB
        API["api.py\nglobal_config.py\nbench_*.py\ncheck_env.py"]:::python
        LaunchServer["launch_server.py"]:::python

        subgraph "Language Compiler/Interpreter"
            direction TB
            LangComp["compiler.py\ninterpreter.py\nir.py\ntracer.py\nchat_template.py\nchoices.py"]:::python
            Backend["lang/backend/"]:::python
        end

        subgraph "Inference Engine (srt)"
            direction TB
            EntryPoints["entrypoints/\nEngineBase.py\nengine.py\nhttp_server.py\nverl_engine.py"]:::python
            Connector["connector/"]:::python
            ModelLoader["model_loader/"]:::python
            Managers["managers/"]:::python
            Sampling["sampling/\nspeculative/"]:::python
            Ops["_custom_ops.py\ncustom_op.py"]:::python
            Configs["configs/"]:::python
        end
    end

    %% Native Kernel Library
    subgraph "Native Kernel Library: sgl-kernel"
        direction TB
        Kernels["csrc/"]:::native
        Build["CMakeLists.txt\nbuild.sh"]:::native
        Bindings["python/sgl_kernel/"]:::native
    end

    %% Router Service
    subgraph "Router Service: sgl-router"
        direction TB
        RouterRust["src/"]:::native
        RouterPy["py_src/sglang_router/\nlaunch_router.py\nlaunch_server.py"]:::python
        RouterTest["py_test/"]:::python
    end

    %% External Services
    subgraph "External Services"
        direction TB
        Judge["LLM Judge API"]:::external
        HF["Hugging Face"]:::external
    end

    %% Connections
    CLI1 -->|parses args & invokes| API
    CLI2 -->|parses args & invokes| API
    CLI3 -->|parses args & invokes| API
    CLI4 -->|parses args & invokes| API
    CLI5 -->|parses args & invokes| API
    CLI6 -->|parses args & invokes| API
    CLI7 -->|parses args & invokes| API
    ShellBase --> CLI1
    ShellST --> CLI1

    CLI1 -->|loads data| DataLoad
    CLI1 -->|downloads models| ModelDL

    API -->|compile & plan| LangComp
    LangComp -->|calls| EntryPoints
    Backend -->|provides connectors| LangComp

    EntryPoints -->|dispatch| Ops
    EntryPoints -->|dispatch| Sampling
    EntryPoints -->|dispatch| ModelLoader
    EntryPoints -->|dispatch| Managers

    Ops --> Kernels
    Kernels --> Bindings

    EntryPoints -->|may route| RouterPy
    RouterRust --> RouterPy

    EntryPoints -->|HTTP judge| Judge
    Judge -->|response| EntryPoints

    API -->|invoke server| LaunchServer
    LaunchServer -->|HTTP| EntryPoints

    EntryPoints -->|final results| HF

    %% Click Events
    click CLI1 "https://github.com/eric-ai-lab/soft-thinking/blob/main/run_sglang_softthinking.py"
    click CLI2 "https://github.com/eric-ai-lab/soft-thinking/blob/main/run_sglang_nothinking.py"
    click CLI3 "https://github.com/eric-ai-lab/soft-thinking/blob/main/demo.py"
    click CLI4 "https://github.com/eric-ai-lab/soft-thinking/blob/main/codeeval.py"
    click CLI5 "https://github.com/eric-ai-lab/soft-thinking/blob/main/humanevaleval.py"
    click CLI6 "https://github.com/eric-ai-lab/soft-thinking/blob/main/matheval.py"
    click CLI7 "https://github.com/eric-ai-lab/soft-thinking/blob/main/mbppeval.py"
    click ShellBase "https://github.com/eric-ai-lab/soft-thinking/tree/main/scripts/baseline/"
    click ShellST "https://github.com/eric-ai-lab/soft-thinking/tree/main/scripts/st/"
    click DataLoad "https://github.com/eric-ai-lab/soft-thinking/tree/main/datasets/"
    click ModelDL "https://github.com/eric-ai-lab/soft-thinking/blob/main/models/download.py"
    click API "https://github.com/eric-ai-lab/soft-thinking/blob/main/sglang_soft_thinking_pkg/python/sglang/api.py"
    click LangComp "https://github.com/eric-ai-lab/soft-thinking/blob/main/sglang_soft_thinking_pkg/python/sglang/lang/compiler.py"
    click Backend "https://github.com/eric-ai-lab/soft-thinking/tree/main/sglang_soft_thinking_pkg/python/sglang/lang/backend/"
    click EntryPoints "https://github.com/eric-ai-lab/soft-thinking/blob/main/sglang_soft_thinking_pkg/python/sglang/srt/entrypoints/engine.py"
    click Connector "https://github.com/eric-ai-lab/soft-thinking/tree/main/sglang_soft_thinking_pkg/python/sglang/srt/connector/"
    click ModelLoader "https://github.com/eric-ai-lab/soft-thinking/tree/main/sglang_soft_thinking_pkg/python/sglang/srt/model_loader/"
    click Managers "https://github.com/eric-ai-lab/soft-thinking/tree/main/sglang_soft_thinking_pkg/python/sglang/srt/managers/"
    click Sampling "https://github.com/eric-ai-lab/soft-thinking/tree/main/sglang_soft_thinking_pkg/python/sglang/srt/sampling/"
    click Ops "https://github.com/eric-ai-lab/soft-thinking/blob/main/sglang_soft_thinking_pkg/python/sglang/srt/_custom_ops.py"
    click Kernels "https://github.com/eric-ai-lab/soft-thinking/tree/main/sglang_soft_thinking_pkg/sgl-kernel/csrc/"
    click Build "https://github.com/eric-ai-lab/soft-thinking/blob/main/sglang_soft_thinking_pkg/sgl-kernel/CMakeLists.txt"
    click Bindings "https://github.com/eric-ai-lab/soft-thinking/tree/main/sglang_soft_thinking_pkg/sgl-kernel/python/sgl_kernel/"
    click RouterRust "https://github.com/eric-ai-lab/soft-thinking/tree/main/sglang_soft_thinking_pkg/sgl-router/src/"
    click RouterPy "https://github.com/eric-ai-lab/soft-thinking/blob/main/sglang_soft_thinking_pkg/sgl-router/py_src/sglang_router/launch_router.py"

    %% Styles
    classDef cli fill:#d3d3d3,stroke:#333,stroke-width:1px
    classDef python fill:#add8e6,stroke:#333,stroke-width:1px
    classDef native fill:#90ee90,stroke:#333,stroke-width:1px
    classDef external fill:#ffa500,stroke:#333,stroke-width:1px
```
