# squeezeailab/llmcompiler architecture

> Architecture diagram of the GitHub repository squeezeailab/llmcompiler, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-17). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/squeezeailab/llmcompiler
- Repository: https://github.com/squeezeailab/llmcompiler

## Overview

##### Project type and purpose
- LLMCompiler is a framework and command‐line tool that automates the orchestration of parallel function calls for large language models (LLMs).
- It decomposes user problems into tasks, detects which tasks can run in parallel, and schedules API or local tool invocations to optimize latency, cost, and accuracy.

##### File structure
Top‐level files and folders:
- run_llm_compiler.py & evaluate_results.py – CLI entrypoints for running benchmarks and evaluating outputs
- configs/ – per‐benchmark configuration modules (hotpotqa, movie, parallelqa) and React variants, each defining GPT/LLaMA prompts and tool interfaces
- datasets/ – JSON data for each benchmark
- src/ – core library, subdivided into:
◦ llm_compiler/ – Compiler’s core components (planner, task fetching, output parser, constants)
◦ executors/ – AgentExecutor and schema defining how individual tasks are executed
◦ chains/ – abstractions for LLM call sequences (LLMChain, MathChain)
◦ agents/ – high‐level agent interface, structured chat agent, tools registry
◦ react/ – ReAct baseline implementation
◦ docstore/ – data retrieval (e.g. Wikipedia lookup)
◦ tools/ – base class for user‐provided function calls
◦ callbacks/ – instrumentation hooks for logging, streaming
◦ utils/ – evaluation, logging, model utilities, timing

Key patterns: layered separation of concerns (CLI → configuration → core engine → execution → tooling → external APIs), plugin pattern for benchmarks and tools, streaming pipeline for planner → executor.

##### README insights
- Supports multiple model backends: OpenAI GPT, Azure, Friendli, vLLM
- Configuration objects declare tools (functions + metadata) and in‐context examples per benchmark
- Streaming option enables pipelined execution (planner emits tasks as they’re ready)

##### Main components and their interactions
a. CLI Layer:
- run_llm_compiler.py – parses arguments, loads config, instantiates LLMCompiler, triggers run
- evaluate_results.py – post‐run metrics aggregator
b. Configuration Layer:
- configs/* – each benchmark provides: tool definitions, prompt templates, model settings
c. Core Engine (llm_compiler):
- Planner – receives user question + tools, decomposes into tasks, identifies dependencies
- TaskFetchingUnit – streams task requests to Executor as soon as tasks are generated
- OutputParser – collects results from Executor, merges partial responses
d. Execution Layer:
- AgentExecutor – invokes LLM or tool calls for each task, using Chains and Agents
- Chains – build and send chat/completion payloads to model backend, apply post‐processing
- docstore & Tools – external data lookup (wikipedia), user functions
- Callbacks – progress reporting, logging, streaming responses
e. External Services:
- LLM API endpoints (OpenAI, Azure, Friendli) or local vLLM server
f. Data & Results:
- datasets/ – input examples for each benchmark
- store path (JSON) – writes question, labels, predictions, latency

##### Architectural patterns and design principles
- Modular, layered design with clear separation of CLI, configuration, core orchestration, execution, and I/O
- Plugin architecture for benchmarks and tools
- Streaming pipeline: planner → task fetcher → executor → output parser (enables parallelism)
- Use of callbacks for instrumentation without coupling to core logic

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% CLI Layer
    subgraph "CLI Layer"
        direction TB
        CLI_Run["run_llm_compiler.py"]:::cli
        CLI_Eval["evaluate_results.py"]:::cli
    end

    %% Configuration Layer
    subgraph "Configuration Layer"
        direction TB
        Configs["Benchmark Configs"]:::config
    end

    %% Core Engine
    subgraph "Compiler Core" 
        direction TB
        LLMComp["LLMCompiler (Orchestrator)"]:::core
        Planner["Planner"]:::core
        TaskFetch["TaskFetchingUnit"]:::core
        OutputParse["OutputParser"]:::core
        Constants["Constants"]:::core
    end

    %% Execution Layer
    subgraph "Execution Layer"
        direction TB
        AgentExec["AgentExecutor"]:::exec
        Schema["Executor Schema"]:::exec
        subgraph "Chains"
            direction TB
            ChainBase["chain.py"]:::exec
            LLMChain["llm_chain.py"]:::exec
            MathChain["llm_math_chain.py"]:::exec
        end
        subgraph "Agents Interface"
            direction TB
            AgentBase["agent.py"]:::exec
            StructuredAgent["structured_chat_agent.py"]:::exec
            ToolsReg["tools.py"]:::exec
            TypesDef["types.py"]:::exec
        end
        ReactBase["react/base.py"]:::exec
        ReactOut["react/output_parser.py"]:::exec
    end

    %% Tools, Docstore, Callbacks, Utils
    subgraph "Support Services"
        direction TB
        Docstore["wikipedia.py"]:::ext
        ToolsBase["base.py"]:::exec
        Callbacks["callbacks.py"]:::exec
        subgraph "Utilities"
            direction TB
            EvalUtil["evaluation_utils.py"]:::util
            LoggerUtil["logger_utils.py"]:::util
            ModelUtil["model_utils.py"]:::util
            TimeUtil["time_utils.py"]:::util
        end
    end

    %% Data & External
    subgraph "Data & External"
        direction TB
        Datasets["datasets/"]:::data
        ResultsStore["Results Store (JSON)"]:::data
        subgraph "LLM API Endpoints"
            direction TB
            OpenAI["OpenAI API"]:::external
            Azure["Azure API"]:::external
            Friendli["Friendli API"]:::external
            vLLM["vLLM Server"]:::external
        end
    end

    %% Connections
    CLI_Run -->|"load config"| Configs
    Configs -->|"init"| LLMComp
    CLI_Run -->|"start"| LLMComp
    LLMComp -->|"use constants"| Constants
    LLMComp -->|"decompose"| Planner
    Planner -->|"stream tasks"| TaskFetch
    TaskFetch -->|"dispatch"| AgentExec
    AgentExec --> Schema
    AgentExec --> ChainBase
    AgentExec --> LLMChain
    AgentExec --> MathChain
    AgentExec --> AgentBase
    AgentExec --> StructuredAgent
    AgentExec --> ToolsReg
    AgentExec --> TypesDef
    AgentExec --> ReactBase
    AgentExec --> ReactOut
    AgentExec -->|"API calls"| OpenAI
    AgentExec -->|"API calls"| Azure
    AgentExec -->|"API calls"| Friendli
    AgentExec -->|"API calls"| vLLM
    AgentExec -->|"retrieve data"| Docstore
    AgentExec -->|"invoke tool"| ToolsBase
    AgentExec -->|"report"| Callbacks
    AgentExec -->|"log/time"| EvalUtil
    AgentExec -->|"log/time"| LoggerUtil
    AgentExec -->|"model helpers"| ModelUtil
    AgentExec -->|"timing"| TimeUtil
    AgentExec -->|"results"| OutputParse
    OutputParse -->|"write"| ResultsStore
    CLI_Eval -->|"analyze"| ResultsStore
    Datasets -->|"input examples"| Configs

    %% Click events
    click CLI_Run "https://github.com/squeezeailab/llmcompiler/blob/main/run_llm_compiler.py"
    click CLI_Eval "https://github.com/squeezeailab/llmcompiler/blob/main/evaluate_results.py"
    click Configs "https://github.com/squeezeailab/llmcompiler/tree/main/configs/"
    click Planner "https://github.com/squeezeailab/llmcompiler/blob/main/src/llm_compiler/planner.py"
    click TaskFetch "https://github.com/squeezeailab/llmcompiler/blob/main/src/llm_compiler/task_fetching_unit.py"
    click OutputParse "https://github.com/squeezeailab/llmcompiler/blob/main/src/llm_compiler/output_parser.py"
    click LLMComp "https://github.com/squeezeailab/llmcompiler/blob/main/src/llm_compiler/llm_compiler.py"
    click Constants "https://github.com/squeezeailab/llmcompiler/blob/main/src/llm_compiler/constants.py"
    click AgentExec "https://github.com/squeezeailab/llmcompiler/blob/main/src/executors/agent_executor.py"
    click Schema "https://github.com/squeezeailab/llmcompiler/blob/main/src/executors/schema.py"
    click ChainBase "https://github.com/squeezeailab/llmcompiler/blob/main/src/chains/chain.py"
    click LLMChain "https://github.com/squeezeailab/llmcompiler/blob/main/src/chains/llm_chain.py"
    click MathChain "https://github.com/squeezeailab/llmcompiler/blob/main/src/chains/llm_math_chain.py"
    click AgentBase "https://github.com/squeezeailab/llmcompiler/blob/main/src/agents/agent.py"
    click StructuredAgent "https://github.com/squeezeailab/llmcompiler/blob/main/src/agents/structured_chat_agent.py"
    click ToolsReg "https://github.com/squeezeailab/llmcompiler/blob/main/src/agents/tools.py"
    click TypesDef "https://github.com/squeezeailab/llmcompiler/blob/main/src/agents/types.py"
    click ReactBase "https://github.com/squeezeailab/llmcompiler/blob/main/src/react/base.py"
    click ReactOut "https://github.com/squeezeailab/llmcompiler/blob/main/src/react/output_parser.py"
    click Docstore "https://github.com/squeezeailab/llmcompiler/blob/main/src/docstore/wikipedia.py"
    click ToolsBase "https://github.com/squeezeailab/llmcompiler/blob/main/src/tools/base.py"
    click Callbacks "https://github.com/squeezeailab/llmcompiler/blob/main/src/callbacks/callbacks.py"
    click EvalUtil "https://github.com/squeezeailab/llmcompiler/blob/main/src/utils/evaluation_utils.py"
    click LoggerUtil "https://github.com/squeezeailab/llmcompiler/blob/main/src/utils/logger_utils.py"
    click ModelUtil "https://github.com/squeezeailab/llmcompiler/blob/main/src/utils/model_utils.py"
    click TimeUtil "https://github.com/squeezeailab/llmcompiler/blob/main/src/utils/time_utils.py"
    click Datasets "https://github.com/squeezeailab/llmcompiler/tree/main/datasets/"
    
    %% Styles
    classDef cli fill:#e0e0ff,stroke:#8888ff,color:#000;
    classDef config fill:#fff0b3,stroke:#ffd24d,color:#000;
    classDef core fill:#cce5ff,stroke:#3399ff,color:#000;
    classDef exec fill:#e6ffe6,stroke:#33cc33,color:#000;
    classDef util fill:#f2e6ff,stroke:#b84dff,color:#000;
    classDef ext fill:#d9ead3,stroke:#6aa84f,color:#000;
    classDef data fill:#ffe6cc,stroke:#ff9933,color:#000;
    classDef external fill:#d0e0f0,stroke:#5485c5,color:#000;
```
