How eth-sri/lmql works
Main Components
• Core LMQL Library (Python)
– Language Layer: src/lmql/language (compiler, parser, validator)
– Runtime & Interpreter: src/lmql/runtime (execution engine, scheduling, caching, context)
– Operations & Models: src/lmql/ops, src/lmql/models/lmtp (model‐abstraction, backends for OpenAI/Transformers/Replicate)
– API Layer: src/lmql/api (sync/async Python API for running queries and serving)
– Output Streaming: src/lmql/output (REST, WebSocket, SSE)
• Command-Line Tools & Entrypoints
– src/lmql/cli.py, api.py, scripts/serve-all.py
– lmql run, lmql playground, lmql serve-model
• HTTP Inference Server
– Implements REST/SSE/WS endpoints via src/lmql/http.py and src/lmql/output/*
– Used by browser and programmatic clients
• UI Frontends
– Browser-based Playground (src/lmql/ui/playground): React/Node app talking over HTTP/WS to the inference API
– Interactive Chat UI (src/lmql/lib/chat): lightweight web UI served by the Python package
– VSCode Extension (src/lmql/ui/vscode): grammar, syntax highlighting, language‐server stubs
• Documentation Site & Build Tooling
– VitePress docs (docs/.vitepress) and static site
– Build scripts (scripts/browser-build, Dockerfiles, Nix/flock files)
• External Services
– OpenAI/Azure APIs, HuggingFace Transformers, Replicate, LangChain, LlamaIndex integrations
Relationships and Data Flows
• User (writes .lmql or uses CLI/Playground)
→ CLI/Playground UI invokes
→ Core API which calls
→ Language Layer to parse/compile LMQL code into an AST
→ Runtime to execute AST (control-flow, constraint-solving, caching)
→ Model Abstraction (lmtp) to dispatch requests to external backends (OpenAI, Transformers)
→ External LLM APIs return tokens
→ Runtime applies post-processing, streaming via
→ Output Layer (HTTP/SSE/WS or direct CLI stdout)
→ UI Frontend or Client Code displays results
Architectural Patterns
• Layered Architecture: clear separation of parsing, compilation, runtime, and model layers
• Plugin/Integration Points: src/lmql/api, src/lmql/models/lmtp, src/lmql/runtime/openai_integration.py
• Event-Driven Streaming: SSE and WebSocket for incremental token output
• Microservice-Style (optional): HTTP inference server can run standalone in Docker (scripts/deploy-web.sh)
• Monorepo Polyglot: Python core + Node.js frontends + CI/CD GitHub Actions