# explosion/spacy-llm architecture

> Architecture diagram of the GitHub repository explosion/spacy-llm, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-12). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/explosion/spacy-llm
- Repository: https://github.com/explosion/spacy-llm

## Overview

##### Project Type and Purpose
This is an open-source Python library (“spacy-llm”) that extends spaCy by integrating Large Language Models (LLMs) into the spaCy pipeline. It is neither a standalone service nor a web app, but a modular toolkit for fast prototyping of NLP tasks using prompts and LLMs (OpenAI, Cohere, Anthropic, Google PaLM, Azure, Hugging Face models, LangChain, etc.).

##### File‐Structure Highlights
- Top‐level files: pyproject.toml, setup.py, README.md, migration guide, CI/workflow configs
- Core package: `spacy_llm/`
- `pipeline/llm.py`: spaCy pipeline component definition
- `models/`: adapter packages for local HF models, LangChain, REST APIs (OpenAI, cohere, …), plus `base.py` and `cache.py`
- `tasks/`: prompt templates, parsers, utilities for each NLP task (ner, textcat, rel, span, summarization, translation, etc.)
- `registry/`: spaCy registry entries for pluggable tasks, models, normalizers, readers
- `util.py`, `compat.py`, `cache.py`: shared utilities
- Usage examples: show run_pipeline.py for various tasks and backends
- Tests: thorough test suites for models, pipeline, tasks, sharding, examples

##### Architectural Overview
- spaCy (host)
- The `llm` pipeline component (in your custom pipeline)
- Task modules (prompt construction + parsing logic)
- Model adapters (REST clients, HF inference, LangChain wrappers)
- Cache layer (to memoize LLM calls)
- Registry (extension mechanism)
- External LLM services & local model engines

##### Main Components & Relationships
a. spaCy Pipeline Component (spacy_llm.pipeline.llm)
- Entry point in spaCy’s pipeline.
- Receives a spaCy Doc → delegates to a selected “Task.”

b. Task Layer (spacy_llm.tasks)
- Defines the interface: build prompt(s) from Doc → post‐process LLM output into spacy data structures (entities, labels, spans).
- Contains per‐task submodules (ner, textcat, summarization, etc.) with Jinja templates in `tasks/templates`.

c. Model Layer (spacy_llm.models)
- Provides a unified `.generate(prompt)` interface.
- Subpackages:
– `models/rest/…`: REST API clients for OpenAI, Cul, Cohere, Azure, Palm, Anthropic, NoOp
– `models/hf/…`: Hugging Face local inference (Falcon, Dolly, Llama2, Mistral, etc.)
– `models/langchain`: wraps LangChain LLM classes
- `cache.py` intercepts calls to avoid duplicate API requests.

d. Registry (spacy_llm.registry)
- spaCy’s registry entries let users register custom tasks, models, prompt normalizers, readers.

e. External Services / Engines
- OpenAI, Cohere, Anthropic, Google PaLM, Microsoft Azure, etc.
- Local HF model runtime (Torch/Transformers)
- LangChain orchestration

##### Key Architectural Patterns
- Adapter/Strategy Pattern in Model Layer: abstracts each LLM provider behind a common interface.
- Plugin/Registry Pattern: spaCy registry for tasks, models, normalizers, readers.
- Pipeline Pattern: spaCy pipeline component orchestrates tasks + models + caching.
- Template & Parser Pattern: Jinja templates + parsing utilities.
- ‐Reduce (Shard) Pattern: splitting prompts too large for context windows, then merging results.

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Input and Output
    DocIn["Input Text\n(spaCy Doc)"]
    DocOut["Enriched Doc\n(entities, labels, spans)"]

    %% Core Pipeline Component
    subgraph "Core Library Modules"
        direction TB
        llmComp["spacy_llm.pipeline.llm"]:::core
        util["spacy_llm/util.py"]:::core
        compat["spacy_llm/compat.py"]:::core
        cacheRoot["Cache Layer\n(spacy_llm/cache.py)"]:::core
    end

    %% Task Layer
    subgraph "Task Layer" 
        direction TB
        tasksDir["spacy_llm/tasks/"]:::core
        builtinTask["builtin_task.py"]:::core
        nerTask["NER Task\n(task.py)"]:::core
        textcatTask["TextCat Task\n(task.py)"]:::core
        summarTask["Summarization Task\n(task.py)"]:::core
        templates["Templates\n(spacy_llm/tasks/templates/)"]:::core
    end

    %% Model Layer
    subgraph "Model Layer"
        direction TB
        modelBase["Base Interface\n(spacy_llm/models/base.py)"]:::core
        restAdapters["REST Adapters\n(spacy_llm/models/rest/)"]:::core
        hfAdapters["HuggingFace Adapters\n(spacy_llm/models/hf/)"]:::core
        lcAdapter["LangChain Wrapper\n(model.py)"]:::core
    end

    %% Registry & Extensions
    subgraph "Registry & Extensions"
        direction TB
        registryDir["Registry\n(spacy_llm/registry/)"]:::registry
        userExt["User-defined"]:::registry
    end

    %% External Services
    subgraph "External LLM Services"
        direction TB
        extCloud["OpenAI, Cohere, Anthropic,\nGoogle PaLM, Azure, ..."]:::external
        hfRuntime["Local HF Runtime\n(Torch/Transformers)"]:::external
    end

    %% Flows
    DocIn -->|"Doc →"| llmComp
    llmComp -->|"delegates to"| tasksDir
    tasksDir -->|"builds prompt"| cacheRoot
    cacheRoot -->|"cache miss"| restAdapters
    cacheRoot -->|"cache miss"| hfAdapters
    cacheRoot -->|"cache miss"| lcAdapter
    restAdapters -->|"API call"| extCloud
    hfAdapters -->|"inference"| hfRuntime
    lcAdapter -->|"chain"| extCloud
    extCloud -->|"response"| restAdapters
    hfRuntime -->|"response"| hfAdapters
    restAdapters -->|"response"| cacheRoot
    hfAdapters -->|"response"| cacheRoot
    lcAdapter -->|"response"| cacheRoot
    cacheRoot -->|"response"| tasksDir
    tasksDir -->|"parses output"| llmComp
    llmComp -->|"Doc ←"| DocOut

    %% Registry Flows
    userExt -->|"registers with"| registryDir
    registryDir -->|"provides plugin to"| tasksDir
    registryDir -->|"provides plugin to"| restAdapters
    registryDir -->|"provides plugin to"| hfAdapters
    registryDir -->|"provides plugin to"| llmComp

    %% Click Events
    click llmComp "https://github.com/explosion/spacy-llm/blob/main/spacy_llm/pipeline/llm.py"
    click util "https://github.com/explosion/spacy-llm/blob/main/spacy_llm/util.py"
    click compat "https://github.com/explosion/spacy-llm/blob/main/spacy_llm/compat.py"
    click cacheRoot "https://github.com/explosion/spacy-llm/blob/main/spacy_llm/cache.py"
    click tasksDir "https://github.com/explosion/spacy-llm/tree/main/spacy_llm/tasks/"
    click builtinTask "https://github.com/explosion/spacy-llm/blob/main/spacy_llm/tasks/builtin_task.py"
    click nerTask "https://github.com/explosion/spacy-llm/blob/main/spacy_llm/tasks/ner/task.py"
    click textcatTask "https://github.com/explosion/spacy-llm/blob/main/spacy_llm/tasks/textcat/task.py"
    click summarTask "https://github.com/explosion/spacy-llm/blob/main/spacy_llm/tasks/summarization/task.py"
    click templates "https://github.com/explosion/spacy-llm/tree/main/spacy_llm/tasks/templates/"
    click modelBase "https://github.com/explosion/spacy-llm/blob/main/spacy_llm/models/base.py"
    click restAdapters "https://github.com/explosion/spacy-llm/tree/main/spacy_llm/models/rest/"
    click hfAdapters "https://github.com/explosion/spacy-llm/tree/main/spacy_llm/models/hf/"
    click lcAdapter "https://github.com/explosion/spacy-llm/blob/main/spacy_llm/models/langchain/model.py"
    click registryDir "https://github.com/explosion/spacy-llm/tree/main/spacy_llm/registry/"

    %% Styles
    classDef core fill:#D6EFFF,stroke:#333,stroke-width:1px
    classDef external fill:#E2F0D9,stroke:#333,stroke-width:1px
    classDef registry fill:#FFF2CC,stroke:#333,stroke-width:1px
```
