How explosion/spacy-llm works
Project Type and Purpose
This is an open-source Python library (“spacy-llm”) that extends spaCy by integrating Large Language Models (LLMs) into the spaCy pipeline. It is neither a standalone service nor a web app, but a modular toolkit for fast prototyping of NLP tasks using prompts and LLMs (OpenAI, Cohere, Anthropic, Google PaLM, Azure, Hugging Face models, LangChain, etc.).
File‐Structure Highlights
• Top‐level files: pyproject.toml, setup.py, README.md, migration guide, CI/workflow configs
• Core package: spacy_llm/
• pipeline/llm.py: spaCy pipeline component definition
• models/: adapter packages for local HF models, LangChain, REST APIs (OpenAI, cohere, …), plus base.py and cache.py
• tasks/: prompt templates, parsers, utilities for each NLP task (ner, textcat, rel, span, summarization, translation, etc.)
• registry/: spaCy registry entries for pluggable tasks, models, normalizers, readers
• util.py, compat.py, cache.py: shared utilities
• Usage examples: show run_pipeline.py for various tasks and backends
• Tests: thorough test suites for models, pipeline, tasks, sharding, examples
Architectural Overview
• spaCy (host)
• The llm pipeline component (in your custom pipeline)
• Task modules (prompt construction + parsing logic)
• Model adapters (REST clients, HF inference, LangChain wrappers)
• Cache layer (to memoize LLM calls)
• Registry (extension mechanism)
• External LLM services & local model engines
Main Components & Relationships
a. spaCy Pipeline Component (spacy_llm.pipeline.llm)
• Entry point in spaCy’s pipeline.
• Receives a spaCy Doc → delegates to a selected “Task.”
b. Task Layer (spacy_llm.tasks)
• Defines the interface: build prompt(s) from Doc → post‐process LLM output into spacy data structures (entities, labels, spans).
• Contains per‐task submodules (ner, textcat, summarization, etc.) with Jinja templates in tasks/templates.
c. Model Layer (spacy_llm.models)
• Provides a unified .generate(prompt) interface.
• Subpackages:
– models/rest/…: REST API clients for OpenAI, Cul, Cohere, Azure, Palm, Anthropic, NoOp
– models/hf/…: Hugging Face local inference (Falcon, Dolly, Llama2, Mistral, etc.)
– models/langchain: wraps LangChain LLM classes
• cache.py intercepts calls to avoid duplicate API requests.
d. Registry (spacy_llm.registry)
• spaCy’s registry entries let users register custom tasks, models, prompt normalizers, readers.
e. External Services / Engines
• OpenAI, Cohere, Anthropic, Google PaLM, Microsoft Azure, etc.
• Local HF model runtime (Torch/Transformers)
• LangChain orchestration
Key Architectural Patterns
• Adapter/Strategy Pattern in Model Layer: abstracts each LLM provider behind a common interface.
• Plugin/Registry Pattern: spaCy registry for tasks, models, normalizers, readers.
• Pipeline Pattern: spaCy pipeline component orchestrates tasks + models + caching.
• Template & Parser Pattern: Jinja templates + parsing utilities.
• ‐Reduce (Shard) Pattern: splitting prompts too large for context windows, then merging results.