How eric-ai-lab/soft-thinking works
Project Type and Purpose
• This is a research‐oriented LLM inference and evaluation framework (“Soft Thinking”) implementing the techniques from the paper “Soft Thinking: Unlocking the Reasoning Potential of LLMs….”
• It orchestrates data loading, model download, baseline and “soft thinking” inference modes, optional LLM‐based judgments, and result upload to Hugging Face.
Top‐Level Architecture
• CLI / Orchestration Layer
• Root scripts (run_sglang_softthinking.py, run_sglang_nothinking.py, demo.py, codeeval.py, humanevaleval.py, etc.) and shell wrappers in scripts/baseline and scripts/st.
• Responsibilities: parse args, set up environment, invoke core Python package.
• Data Layer
• datasets/: JSON and helper‐py modules load benchmarks (HumanEval, MBPP, AIME, etc.).
• models/download.py: fetches model artifacts into models/.
• Core Inference Package
• sglang_soft_thinking_pkg/python/sglang: the main Python API and orchestration logic.
• Subcomponents:
· lang/: compiler, interpreter, IR, chat templates, backends for OpenAI, Anthropic, VertexAI, LiteLLM, runtime endpoint.
· srt/: high‐performance inference engine, dispatching to CPU/GPU/triton/ROCm kernels, quantization, MoE, speculative sampling, multi‐GPU distributed.
· launch_server.py / launch_router.py: internal HTTP endpoints or router components.
· models/, connector/, managers/, model_loader/: configuration, data parallel/distributed state, memory/cache management.
• Kernel Module
• sgl-kernel: custom C++/CUDA/HIP kernel implementations for attention, GEMM, quantization, MoE, speculative sampling.
• Python bindings (sgl_kernel/python/sgl_kernel) expose ops to the SRT engine.
• Router Service
• sgl-router: Rust‐based routing service for distributing inference requests or coordinating multi‐host setups.
• External Services
• LLM Judge via API (OpenAI/Anthropic/Vertex) for evaluating sample quality.
• Hugging Face (optional) for pushing results.
Relationships & Data Flows
1. User runs a shell script or Python entrypoint.
2. Orchestration layer loads dataset, downloads or locates model.
3. Calls into sglang API, which compiles prompts, sets sampling parameters (soft thinking vs baseline).
4. sglang.srt engine dispatches to the sgl-kernel library (via Python bindings) to execute forward passes, attention, GEMM, MoE, quantization on GPU/CPU.
5. Partial outputs may be sent back to the interpreter for “after thinking” passes.
6. Completed samples can optionally be forwarded to an external LLM judge via HTTP.
7. Final results are collected, evaluated (e.g., correctness metrics), and optionally pushed to Hugging Face.
Architectural Patterns and Principles
• Modular layering: CLI → Python API → Inference engine → Native kernels.
• Plug‐in backends: abstract base classes in lang/backend, concrete connectors for OpenAI, Anthropic, etc.
• Separation of concerns: data loading, model loading, prompt compilation, sampling strategy, kernel execution, result evaluation.
• High-performance: custom kernels (CUDA/HIP), Triton integration, quantization schemes, caching, distributed communication.