How squeezeailab/llmcompiler works
Project type and purpose
• LLMCompiler is a framework and command‐line tool that automates the orchestration of parallel function calls for large language models (LLMs).
• It decomposes user problems into tasks, detects which tasks can run in parallel, and schedules API or local tool invocations to optimize latency, cost, and accuracy.
File structure
Top‐level files and folders:
• run_llm_compiler.py & evaluate_results.py – CLI entrypoints for running benchmarks and evaluating outputs
• configs/ – per‐benchmark configuration modules (hotpotqa, movie, parallelqa) and React variants, each defining GPT/LLaMA prompts and tool interfaces
• datasets/ – JSON data for each benchmark
• src/ – core library, subdivided into:
◦ llm_compiler/ – Compiler’s core components (planner, task fetching, output parser, constants)
◦ executors/ – AgentExecutor and schema defining how individual tasks are executed
◦ chains/ – abstractions for LLM call sequences (LLMChain, MathChain)
◦ agents/ – high‐level agent interface, structured chat agent, tools registry
◦ react/ – ReAct baseline implementation
◦ docstore/ – data retrieval (e.g. Wikipedia lookup)
◦ tools/ – base class for user‐provided function calls
◦ callbacks/ – instrumentation hooks for logging, streaming
◦ utils/ – evaluation, logging, model utilities, timing
Key patterns: layered separation of concerns (CLI → configuration → core engine → execution → tooling → external APIs), plugin pattern for benchmarks and tools, streaming pipeline for planner → executor.
README insights
• Supports multiple model backends: OpenAI GPT, Azure, Friendli, vLLM
• Configuration objects declare tools (functions + metadata) and in‐context examples per benchmark
• Streaming option enables pipelined execution (planner emits tasks as they’re ready)
Main components and their interactions
a. CLI Layer:
• run_llm_compiler.py – parses arguments, loads config, instantiates LLMCompiler, triggers run
• evaluate_results.py – post‐run metrics aggregator
b. Configuration Layer:
• configs/* – each benchmark provides: tool definitions, prompt templates, model settings
c. Core Engine (llm_compiler):
• Planner – receives user question + tools, decomposes into tasks, identifies dependencies
• TaskFetchingUnit – streams task requests to Executor as soon as tasks are generated
• OutputParser – collects results from Executor, merges partial responses
d. Execution Layer:
• AgentExecutor – invokes LLM or tool calls for each task, using Chains and Agents
• Chains – build and send chat/completion payloads to model backend, apply post‐processing
• docstore & Tools – external data lookup (wikipedia), user functions
• Callbacks – progress reporting, logging, streaming responses
e. External Services:
• LLM API endpoints (OpenAI, Azure, Friendli) or local vLLM server
f. Data & Results:
• datasets/ – input examples for each benchmark
• store path (JSON) – writes question, labels, predictions, latency
Architectural patterns and design principles
• Modular, layered design with clear separation of CLI, configuration, core orchestration, execution, and I/O
• Plugin architecture for benchmarks and tools
• Streaming pipeline: planner → task fetcher → executor → output parser (enables parallelism)
• Use of callbacks for instrumentation without coupling to core logic