# eth-sri/lmql architecture

> Architecture diagram of the GitHub repository eth-sri/lmql, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-11-10). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/eth-sri/lmql
- Repository: https://github.com/eth-sri/lmql

## Overview

##### Main Components
- Core LMQL Library (Python)
– **Language Layer**: `src/lmql/language` (compiler, parser, validator)
– **Runtime & Interpreter**: `src/lmql/runtime` (execution engine, scheduling, caching, context)
– **Operations & Models**: `src/lmql/ops`, `src/lmql/models/lmtp` (model‐abstraction, backends for OpenAI/Transformers/Replicate)
– **API Layer**: `src/lmql/api` (sync/async Python API for running queries and serving)
– **Output Streaming**: `src/lmql/output` (REST, WebSocket, SSE)
- Command-Line Tools & Entrypoints
– `src/lmql/cli.py`, `api.py`, `scripts/serve-all.py`
– `lmql run`, `lmql playground`, `lmql serve-model`
- HTTP Inference Server
– Implements REST/SSE/WS endpoints via `src/lmql/http.py` and `src/lmql/output/*`
– Used by browser and programmatic clients
- UI Frontends
– **Browser-based Playground** (`src/lmql/ui/playground`): React/Node app talking over HTTP/WS to the inference API
– **Interactive Chat UI** (`src/lmql/lib/chat`): lightweight web UI served by the Python package
– **VSCode Extension** (`src/lmql/ui/vscode`): grammar, syntax highlighting, language‐server stubs
- Documentation Site & Build Tooling
– VitePress docs (`docs/.vitepress`) and static site
– Build scripts (`scripts/browser-build`, Dockerfiles, Nix/flock files)
- External Services
– OpenAI/Azure APIs, HuggingFace Transformers, Replicate, LangChain, LlamaIndex integrations

##### Relationships and Data Flows
- **User** (writes `.lmql` or uses CLI/Playground)
→ **CLI/Playground UI** invokes
→ **Core API** which calls
→ **Language Layer** to parse/compile LMQL code into an AST
→ **Runtime** to execute AST (control-flow, constraint-solving, caching)
→ **Model Abstraction** (`lmtp`) to dispatch requests to external backends (OpenAI, Transformers)
→ **External LLM APIs** return tokens
→ **Runtime** applies post-processing, streaming via
→ **Output Layer** (HTTP/SSE/WS or direct CLI stdout)
→ **UI Frontend** or **Client Code** displays results

##### Architectural Patterns
- **Layered Architecture**: clear separation of parsing, compilation, runtime, and model layers
- **Plugin/Integration Points**: `src/lmql/api`, `src/lmql/models/lmtp`, `src/lmql/runtime/openai_integration.py`
- **Event-Driven Streaming**: SSE and WebSocket for incremental token output
- **Microservice-Style** (optional): HTTP inference server can run standalone in Docker (`scripts/deploy-web.sh`)
- **Monorepo Polyglot**: Python core + Node.js frontends + CI/CD GitHub Actions

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Clients & UIs
    subgraph "Clients & UIs"
        direction TB
        Playground["Browser Playground"]:::ui
        ChatUI["Interactive Chat UI"]:::ui
        VSCodeExt["VSCode Extension"]:::ui
        CLIClient["CLI"]:::ui
    end

    %% API & Serving Layer
    subgraph "API & Serving Layer"
        direction TB
        HTTPServer["HTTP Inference Server"]:::api
        CLIEntry["CLI Entrypoints"]:::api
    end

    %% Core Library
    subgraph "Core Library (Python)" 
        direction TB

        subgraph "API Layer"
            direction TB
            RunAPI["run.py"]:::core
            ServeAPI["serve.py"]:::core
            QueriesAPI["queries.py"]:::core
            QueryBuilder["query_builder.py"]:::core
            LLMAPI["llm.py"]:::core
            InspectAPI["inspect.py"]:::core
        end

        subgraph "Language Layer"
            direction TB
            Compiler["compiler.py"]:::core
            Parser["fragment_parser.py"]:::core
            Validator["validator.py"]:::core
            QStrings["qstrings.py"]:::core
        end

        subgraph "Runtime & Interpreter"
            direction TB
            Interpreter["interpreter.py"]:::core
            RuntimeCore["lmql_runtime.py"]:::core
            OpenAIInt["openai_integration.py"]:::core
            BatchedOpenAI["batched_openai.py"]:::core
            Context["context.py"]:::core
            Caching["caching.py"]:::core
            OutputWriter["output_writer.py"]:::core
            subgraph "Postprocessing & Tokenizers"
                direction TB
                PostProc["postprocessing"]:::core
                Tokenizers["tokenizers"]:::core
            end
        end

        subgraph "Operations & Models"
            direction TB
            subgraph "Operators & Constraints"
                direction TB
                Ops["ops.py"]:::core
                InlineCall["inline_call.py"]:::core
                RegexOp["regex.py"]:::core
                TokenSet["token_set.py"]:::core
            end
            subgraph "Model Abstraction & Backends"
                direction TB
                TransformersModel["transformers_model.py"]:::core
                LlamaCPP["llama_cpp_model.py"]:::core
                LMTPAsync["lmtp_async.py"]:::core
                LMTPServer["lmtp_inference_server.py"]:::core
                ModelInfo["model_info.py"]:::core
            end
        end

        subgraph "Output Streaming"
            direction TB
            HTTPOut["http.py"]:::core
            SSEOut["sse.py"]:::core
            WSOut["ws.py"]:::core
        end
    end

    %% External Systems
    subgraph "External Systems" 
        direction TB
        OpenAIExt["OpenAI/Azure APIs"]:::external
        HFTransformers["HuggingFace Transformers"]:::external
        ReplicateExt["Replicate"]:::external
        LangChainExt["LangChain"]:::external
        LlamaIndexExt["LlamaIndex"]:::external
    end

    %% Tools & Documentation
    subgraph "Docs & Build Tools"
        direction TB
        Docs["VitePress Docs"]:::tools
        BuildScripts["Build & CI Scripts"]:::tools
    end

    %% Connections
    Playground -->|HTTP/WS| HTTPServer
    ChatUI -->|HTTP/WS| HTTPServer
    VSCodeExt -->|Language Server| Parser
    CLIClient -->|CLI Command| CLIEntry

    HTTPServer -->|calls| ServeAPI
    CLIEntry -->|imports| RunAPI

    ServeAPI -->|executes| Compiler
    RunAPI -->|executes| Compiler

    Compiler -->|AST| RuntimeCore
    Parser -->|AST| RuntimeCore
    Validator -->|validates| RuntimeCore

    RuntimeCore -->|dispatch| OpenAIInt
    RuntimeCore -->|dispatch| BatchedOpenAI
    RuntimeCore -->|uses| Context
    RuntimeCore -->|uses| Caching

    OpenAIInt -->|API Calls| OpenAIExt
    BatchedOpenAI -->|API Calls| HFTransformers

    RuntimeCore -->|postprocess| PostProc
    PostProc -->|format| OutputWriter

    OutputWriter -->|stream| HTTPOut
    OutputWriter -->|stream| SSEOut
    OutputWriter -->|stream| WSOut

    HTTPOut -->|HTTP/JSON| Playground
    SSEOut -->|SSE| Playground
    WSOut -->|WebSocket| Playground
    OutputWriter -->|stdout| CLIClient

    %% Model abstraction branch
    RuntimeCore -->|model| TransformersModel
    RuntimeCore -->|model| LlamaCPP
    TransformersModel -->|inherits| LMTPAsync
    LlamaCPP -->|inherits| LMTPAsync
    LMTPAsync -->|serves| LMTPServer
    LMTPServer -->|API| HTTPServer

    %% Peripheral docs & tools
    Docs --- BuildScripts

    %% Click Events
    click Compiler "https://github.com/eth-sri/lmql/blob/main/src/lmql/language/compiler.py"
    click Parser "https://github.com/eth-sri/lmql/blob/main/src/lmql/language/fragment_parser.py"
    click Validator "https://github.com/eth-sri/lmql/blob/main/src/lmql/language/validator.py"
    click QStrings "https://github.com/eth-sri/lmql/blob/main/src/lmql/language/qstrings.py"
    click Interpreter "https://github.com/eth-sri/lmql/blob/main/src/lmql/runtime/interpreter.py"
    click RuntimeCore "https://github.com/eth-sri/lmql/blob/main/src/lmql/runtime/lmql_runtime.py"
    click OpenAIInt "https://github.com/eth-sri/lmql/blob/main/src/lmql/runtime/openai_integration.py"
    click BatchedOpenAI "https://github.com/eth-sri/lmql/blob/main/src/lmql/runtime/bopenai/batched_openai.py"
    click Context "https://github.com/eth-sri/lmql/blob/main/src/lmql/runtime/context.py"
    click Caching "https://github.com/eth-sri/lmql/blob/main/src/lmql/runtime/caching.py"
    click OutputWriter "https://github.com/eth-sri/lmql/blob/main/src/lmql/runtime/output_writer.py"
    click PostProc "https://github.com/eth-sri/lmql/tree/main/src/lmql/runtime/postprocessing/"
    click Tokenizers "https://github.com/eth-sri/lmql/tree/main/src/lmql/runtime/tokenizers/"
    click Ops "https://github.com/eth-sri/lmql/blob/main/src/lmql/ops/ops.py"
    click InlineCall "https://github.com/eth-sri/lmql/blob/main/src/lmql/ops/inline_call.py"
    click RegexOp "https://github.com/eth-sri/lmql/blob/main/src/lmql/ops/regex.py"
    click TokenSet "https://github.com/eth-sri/lmql/blob/main/src/lmql/ops/token_set.py"
    click TransformersModel "https://github.com/eth-sri/lmql/blob/main/src/lmql/models/lmtp/backends/transformers_model.py"
    click LlamaCPP "https://github.com/eth-sri/lmql/blob/main/src/lmql/models/lmtp/backends/llama_cpp_model.py"
    click LMTPAsync "https://github.com/eth-sri/lmql/blob/main/src/lmql/models/lmtp/lmtp_async.py"
    click LMTPServer "https://github.com/eth-sri/lmql/blob/main/src/lmql/models/lmtp/lmtp_inference_server.py"
    click ModelInfo "https://github.com/eth-sri/lmql/blob/main/src/lmql/models/model_info.py"
    click RunAPI "https://github.com/eth-sri/lmql/blob/main/src/lmql/api/run.py"
    click ServeAPI "https://github.com/eth-sri/lmql/blob/main/src/lmql/api/serve.py"
    click QueriesAPI "https://github.com/eth-sri/lmql/blob/main/src/lmql/api/queries.py"
    click QueryBuilder "https://github.com/eth-sri/lmql/blob/main/src/lmql/api/query_builder.py"
    click LLMAPI "https://github.com/eth-sri/lmql/blob/main/src/lmql/api/llm.py"
    click InspectAPI "https://github.com/eth-sri/lmql/blob/main/src/lmql/api/inspect.py"
    click HTTPOut "https://github.com/eth-sri/lmql/blob/main/src/lmql/output/http.py"
    click SSEOut "https://github.com/eth-sri/lmql/blob/main/src/lmql/output/sse.py"
    click WSOut "https://github.com/eth-sri/lmql/blob/main/src/lmql/output/ws.py"
    click CLIEntry "https://github.com/eth-sri/lmql/blob/main/src/lmql/cli.py"
    click HTTPServer "https://github.com/eth-sri/lmql/blob/main/src/lmql/http.py"
    click Playground "https://github.com/eth-sri/lmql/blob/main/src/lmql/ui/playground/src/App.jsx"
    click Playground "https://github.com/eth-sri/lmql/blob/main/src/lmql/ui/playground/src/index.js"
    click Playground "https://github.com/eth-sri/lmql/blob/main/src/lmql/ui/playground/public/index.html"
    click Playground "https://github.com/eth-sri/lmql/blob/main/src/lmql/ui/playground/package.json"
    click ChatUI "https://github.com/eth-sri/lmql/blob/main/src/lmql/lib/chat/chatserver.py"
    click ChatUI "https://github.com/eth-sri/lmql/blob/main/src/lmql/lib/chat/output.py"
    click ChatUI "https://github.com/eth-sri/lmql/blob/main/src/lmql/lib/chat/assets/index.html"
    click VSCodeExt "https://github.com/eth-sri/lmql/blob/main/src/lmql/ui/vscode/package.json"
    click VSCodeExt "https://github.com/eth-sri/lmql/blob/main/src/lmql/ui/vscode/syntaxes/lmql.tmLanguage.json"
    click VSCodeExt "https://github.com/eth-sri/lmql/blob/main/src/lmql/ui/vscode/language-configuration.json"
    click Docs "https://github.com/eth-sri/lmql/tree/main/docs/.vitepress/"
    click BuildScripts "https://github.com/eth-sri/lmql/tree/main/scripts/browser-build/"
    click BuildScripts "https://github.com/eth-sri/lmql/tree/main/scripts/flake.d/"
    click BuildScripts "https://github.com/eth-sri/lmql/blob/main/Dockerfile.tests"
    click BuildScripts "https://github.com/eth-sri/lmql/tree/main/.github/workflows/"

    %% Styles
    classDef core fill:#D0E8FF,stroke:#333,stroke-width:1px
    classDef ui fill:#DFFFD0,stroke:#333,stroke-width:1px
    classDef external fill:#FFD8A8,stroke:#333,stroke-width:1px
    classDef tools fill:#E8E8E8,stroke:#333,stroke-width:1px
    classDef api fill:#BFE5D7,stroke:#333,stroke-width:1px
```
