# dylanhogg/llmgraph architecture

> Architecture diagram of the GitHub repository dylanhogg/llmgraph, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-10). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/dylanhogg/llmgraph
- Repository: https://github.com/dylanhogg/llmgraph

## Overview

##### Project Type and Purpose
- llmgraph is a Python-based CLI tool and library that automates creation of knowledge graphs from Wikipedia seed pages by invoking large language models (LLMs).
- It ingests a seed entity type and Wikipedia URL, recursively queries an LLM (via LiteLLM/OpenAI), fetches Wikipedia metadata, deduplicates nodes, builds a graph in memory (NetworkX), and exports to GraphML, GEXF, and interactive HTML (pyvis).

##### High-Level Architecture Components
A. CLI Layer
- llmgraph/console.py – parses user flags and options (Click)
- Entrypoint “llmgraph” command
- Reads prompts.yaml and .env for configuration

B. Core Library
- llmgraph/library/engine.py – orchestrator: loop over levels, merge nodes, track tokens
- llmgraph/library/llm.py – adapter to LiteLLM/OpenAI, handles prompt templating, requests, and responses
- llmgraph/library/wikipedia.py – fetches and parses Wikipedia pages (requests/BeautifulSoup)
- llmgraph/library/classes.py – in-memory graph models (Node, Edge, Graph container)
- llmgraph/library/utils.py – helpers (formatting, JSON extraction, file I/O)
- llmgraph/library/prompts.py – loads and renders prompt templates from prompts.yaml
- llmgraph/library/env.py – environment variables and API key management
- llmgraph/library/log.py – structured logging of steps and token usage
- llmgraph/library/consts.py – shared constants (cache paths, default levels, formats)

C. Cache Layer
- .joblib_cache/ – caches LLM and Wikipedia API calls to avoid duplication
- llmgraph/library/env.py and engine.py integrate cache lookups

D. Graph Builder & Exporters
- Graph assembly in memory (NetworkX)
- Exporters:
– GraphML/GEXF via networkx.write_*
– HTML via pyvis (llmgraph/library/utils.py)

E. External Services
- LiteLLM/OpenAI endpoints – LLM model inference
- Wikipedia API or page HTML – seed and supplemental metadata

F. Packaging, Testing & CI
- pyproject.toml/requirements.txt – dependency declaration (Click, LiteLLM, networkx, pyvis)
- tests/ – unit tests for console, JSON extraction, networkx integration, prompts, Wikipedia fetches
- .github/workflows/python-poetry-app.yml – CI for linting, tests, packaging
- pre-commit hooks (.pre-commit-config.yaml)

##### Relationships and Data Flow
1. User invokes `llmgraph <entity_type> <wikipedia_url> [--flags]`
2. console.py parses input → instantiates Engine
3. Engine at level N:
a. Calls prompts.py to render LLM prompt template with entity_root, top_n, etc.
b. llm.py checks cache → if miss, calls LLM API → stores in cache
c. wikipedia.py checks cache → if miss, fetches Wikipedia content → stores in cache
d. classes.py/engine.py create Node and Edge objects, merge duplicates
4. Repeat for deeper levels until `--levels` reached
5. Engine hands final NetworkX graph to utils.py exporters
6. utils.py writes GraphML/GEXF to disk, spins up pyvis for HTML output
7. console.py prints summary (total tokens, output locations)

##### Architectural Patterns & Principles
- Separation of Concerns: CLI, orchestration, LLM abstraction, Wikipedia abstraction, graph model, export
- Caching Decorator/Repository pattern for LLM and Wikipedia calls
- Template Method pattern for prompts rendering
- Dependency Injection of LLM provider via llm-base-url and llm-model flags
- Modular exporters allow additional formats in future

##### Detail
- Caching layer’s role in performance and resumability.
- Modularity that allows swapping out LLM providers.
- Flow of “total tokens” metric from LLM Adapter back to Engine and CLI output.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% CLI Layer
    subgraph "CLI Layer"
        UserCLI["User CLI<br>(llmgraph/console.py)<br>Flags: --levels, --llm-model, --output-folder"]:::internal
    end

    %% Core Library
    subgraph "Core Library"
        Engine["Engine Orchestrator<br>(llmgraph/library/engine.py)"]:::internal
        Prompt["Prompt Renderer<br>(llmgraph/library/prompts.py<br>llmgraph/prompts.yaml)"]:::internal
        LLM["LLM Service Adapter<br>(llmgraph/library/llm.py)"]:::internal
        Wiki["Wikipedia Fetcher<br>(llmgraph/library/wikipedia.py)"]:::internal
        Model["Graph Model<br>(llmgraph/library/classes.py)"]:::internal
        Env["Env & Config<br>(llmgraph/library/env.py)"]:::internal
        Log["Logging & Metrics<br>(llmgraph/library/log.py)"]:::internal
        Const["Constants<br>(llmgraph/library/consts.py)"]:::internal
    end

    %% Cache Layer
    subgraph "Cache Layer"
        Cache["Local Cache<br>(.joblib_cache)"]:::cache
    end

    %% Exporters
    subgraph "Graph Builder & Exporters"
        Export["Exporters & Helpers<br>(llmgraph/library/utils.py)"]:::internal
        Output1["GraphML"]:::output
        Output2["GEXF"]:::output
        Output3["HTML (pyvis)"]:::output
    end

    %% External Services
    subgraph "External Services"
        LLMExt["LLM Endpoint<br>(OpenAI/LiteLLM)"]:::external
        WikiExt["Wikipedia API"]:::external
    end

    %% Packaging & CI
    subgraph "Packaging & CI"
        Pkg["pyproject.toml<br>requirements.txt<br>.github/workflows/python-poetry-app.yml<br>.pre-commit-config.yaml"]:::internal
        Tests["tests/"]:::internal
    end

    %% Flow
    UserCLI -->|"invoke command"| Engine
    Engine -->|orchestrate levels| Prompt
    Prompt -->|"render prompts"| LLM
    LLM -->|check cache| Cache
    Cache -.->|hit/miss| LLM
    LLM -->|"HTTP REST"| LLMExt
    Engine -->|fetch pages| Wiki
    Wiki -->|check cache| Cache
    Cache -.->|hit/miss| Wiki
    Wiki -->|"HTTP/S"| WikiExt
    Prompt -->|provide templates| Model
    LLM -->|generate nodes| Model
    Wiki -->|extract data| Model
    Engine -->|merge duplicates| Model
    Model -->|output graph| Export
    Export -->|write| Output1
    Export -->|write| Output2
    Export -->|render| Output3
    Engine -->|token metrics| Log
    Log -->|report| UserCLI

    %% Click Events
    click UserCLI "https://github.com/dylanhogg/llmgraph/blob/main/llmgraph/console.py"
    click Engine "https://github.com/dylanhogg/llmgraph/blob/main/llmgraph/library/engine.py"
    click Prompt "https://github.com/dylanhogg/llmgraph/blob/main/llmgraph/library/prompts.py"
    click Prompt "https://github.com/dylanhogg/llmgraph/blob/main/llmgraph/prompts.yaml"
    click LLM "https://github.com/dylanhogg/llmgraph/blob/main/llmgraph/library/llm.py"
    click Wiki "https://github.com/dylanhogg/llmgraph/blob/main/llmgraph/library/wikipedia.py"
    click Model "https://github.com/dylanhogg/llmgraph/blob/main/llmgraph/library/classes.py"
    click Export "https://github.com/dylanhogg/llmgraph/blob/main/llmgraph/library/utils.py"
    click Env "https://github.com/dylanhogg/llmgraph/blob/main/llmgraph/library/env.py"
    click Log "https://github.com/dylanhogg/llmgraph/blob/main/llmgraph/library/log.py"
    click Const "https://github.com/dylanhogg/llmgraph/blob/main/llmgraph/library/consts.py"
    click Pkg "https://github.com/dylanhogg/llmgraph/blob/main/pyproject.toml"
    click Pkg "https://github.com/dylanhogg/llmgraph/blob/main/requirements.txt"
    click Pkg "https://github.com/dylanhogg/llmgraph/blob/main/.github/workflows/python-poetry-app.yml"
    click Pkg "https://github.com/dylanhogg/llmgraph/blob/main/.pre-commit-config.yaml"
    click Tests "https://github.com/dylanhogg/llmgraph/tree/main/tests/"

    %% Styles
    classDef internal fill:#BBDEFB,stroke:#1E88E5,color:#0D47A1;
    classDef cache fill:#C8E6C9,stroke:#388E3C,color:#1B5E20;
    classDef external fill:#FFE0B2,stroke:#FB8C00,color:#E65100;
    classDef output fill:#E1BEE7,stroke:#8E24AA,color:#4A148C;
```
