# public-repositories/spacy architecture

> Architecture diagram of the GitHub repository public-repositories/spacy, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-12). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/public-repositories/spacy
- Repository: https://github.com/public-repositories/spacy

## Overview

##### Project type and purpose
- Project: spaCy – an industrial-strength, open-source Natural Language Processing library written in Python/Cython.
- Purpose: Provide a fast, extensible NLP pipeline (tokenization, tagging, parsing, NER, text classification, etc.), a production-ready training system, CLI tooling, and a companion documentation website.

##### Main components
a. Python API Layer (“spacy” package root)
- Entry point for users (`import spacy`, `nlp = spacy.load()`).
- CLI wrappers (`spacy/cli/*`), project templates (`spacy/cli/project/*`).
b. Pipeline Components
- Tokenizer, tagger, parser, entity recognizer, text categorizer, lemmatizer, etc. (`spacy/pipeline/*`).
- Each component implements a `pipe` interface; built-in and legacy versions coexist.
- Extensibility hooks: custom components, factory registration, language data.
c. Core Engine & Cython/C-extensions
- Performance-critical code in Cython (`*.pyx`, `*.pxd` under `spacy/*`, `spacy/pipeline/_parser_internals`, `spacy/matcher/*.pyx`).
- Data structures: `Doc`, `Token`, `Vocab`, `StringStore`, `Vectors`.
d. Model & Data Management
- Pretrained pipelines loaded from disk as Python packages.
- Lookups and language data (lemmatization tables, tokenization exceptions).
e. Training System
- Trainer loop and batchers (`spacy/training/*`), converters for various data formats.
- Callbacks, logging, alignment utilities, multi-task training.
- Project management (`spacy/cli/project`), DVC integration, remote storage.
f. External Framework Integration
- Support for PyTorch, TensorFlow, HuggingFace transformers for custom models.
g. Documentation Website (Gatsby/React)
- Source in `website/` dir; static site generator producing spaCy docs and API reference.
h. CI/CD & Dev Tools
- GitHub Actions, Azure Pipelines, pre-commit hooks.
- Build scripts, versioning scripts (`bin/`), testing suite (`spacy/tests/`).

##### Relationships & Data Flows
- User code or CLI → Python API → Pipeline Manager
- Pipeline Manager → sequential calls to components → Cython core (compute-intensive operations)
- Components load resources: model files, lookup tables → I/O subsystem
- Trainer (CLI or Python) → Training modules → Model serialization → Disk
- CLI/project tooling ↔ Remote storage or DVC → retrieve assets, push results
- Website build process reads Markdown + JSON metadata → Gatsby → static HTML

##### Architectural Patterns & Principles
- Pipeline pattern: each text processing component implements a uniform interface.
- Plugin/Factory registry: decouples component construction and configuration.
- Layered design: Python orchestration shell over Cython core for performance.
- Separation of concerns: parsing, tokenization, NER, etc. isolated modules.
- Extensibility: clear APIs for custom components, support for external ML frameworks.

##### Technologies & Frameworks
- Python 3.6+, Cython for core routines.
- Click/Argparse-style CLI tooling.
- DVC integration for data versioning.
- PyTorch, TensorFlow, HuggingFace for transformer integration.
- Gatsby/React for documentation site.

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% User Interfaces
    subgraph "User Interfaces"
        User["User Code"]:::external
        CLI["CLI Tooling"]:::python
    end

    %% Python API Layer
    subgraph "Python API Layer"
        API["spacy (import & load)"]:::python
    end

    %% Pipeline Manager
    subgraph "Pipeline Manager"
        Manager["Pipeline Runner"]:::python
    end

    %% Pipeline Components
    subgraph "Pipeline Components"
        Tokenizer["Tokenizer"]:::python
        Tagger["Tagger"]:::python
        Parser["Parser"]:::cython
        NER["Entity Recognizer"]:::python
        TextCat["Text Categorizer"]:::python
        Lemmatizer["Lemmatizer"]:::python
    end

    %% Cython Core & Data Structures
    subgraph "Cython Core & Data Structures"
        Doc["Doc, Token, Span"]:::cython
        Vocab["Vocab, StringStore, Vectors"]:::cython
        Matcher["Matcher Internals"]:::cython
    end

    %% Model & Data Management
    subgraph "Model & Data Management"
        ModelFiles["Pretrained Pipelines"]:::storage
        LangData["Language Data & Lookups"]:::storage
    end

    %% Training System
    subgraph "Training System"
        Trainer["Trainer Loop & Batchers"]:::python
        Converters["Data Converters"]:::python
    end

    %% External Frameworks
    subgraph "External Frameworks"
        PyTorch["PyTorch"]:::external
        TensorFlow["TensorFlow"]:::external
        HF["HuggingFace Transformers"]:::external
    end

    %% Documentation Website
    subgraph "Documentation Website"
        Gatsby["Gatsby/React Site"]:::docs
    end

    %% CI/CD & Dev Tools
    subgraph "CI/CD & Dev Tools"
        CI["CI/CD Pipelines"]:::ci
        Scripts["Build & Release Scripts"]:::ci
        Tests["Testing Suite"]:::ci
    end

    %% Flows
    User -->|uses| API
    CLI -->|invokes| API
    API -->|orchestrates| Manager
    Manager -->|calls| Tokenizer
    Manager --> Tagger
    Manager --> Parser
    Manager --> NER
    Manager --> TextCat
    Manager --> Lemmatizer

    Parser -->|C operations| Doc
    Parser --> Vocab
    Parser --> Matcher

    Tokenizer -->|loads| LangData
    Tagger -->|loads| ModelFiles
    NER --> ModelFiles
    TextCat --> ModelFiles
    Lemmatizer --> LangData

    CLI -->|trains| Trainer
    API -->|trains| Trainer
    Trainer --> Converters
    Trainer -->|writes| ModelFiles

    PyTorch -.Integrates with.-> Parser
    TensorFlow -.Integrates with.-> Parser
    HF -.Integrates with.-> Parser

    Gatsby -->|builds site from| website_src["Markdown + JSON"]:::docs
    website_src --> Gatsby

    CI -->|runs| Tests
    CI -->|runs| Scripts
    Scripts -->|versioning| API

    %% Click Events
    click API "https://github.com/public-repositories/spacy/blob/master/spacy/__init__.py"
    click CLI "https://github.com/public-repositories/spacy/tree/master/spacy/cli/"
    click CLI "https://github.com/public-repositories/spacy/tree/master/spacy/cli/project/"
    click CLI "https://github.com/public-repositories/spacy/tree/master/spacy/cli/templates/"
    click Tokenizer "https://github.com/public-repositories/spacy/blob/master/spacy/tokenizer.pyx"
    click Tokenizer "https://github.com/public-repositories/spacy/blob/master/spacy/tokenizer.pxd"
    click Doc "https://github.com/public-repositories/spacy/tree/master/spacy/tokens/"
    click Vocab "https://github.com/public-repositories/spacy/blob/master/spacy/vocab.pyx"
    click Matcher "https://github.com/public-repositories/spacy/tree/master/spacy/matcher/"
    click Parser "https://github.com/public-repositories/spacy/tree/master/spacy/pipeline/_parser_internals/"
    click LangData "https://github.com/public-repositories/spacy/tree/master/spacy/lang/"
    click Converters "https://github.com/public-repositories/spacy/tree/master/spacy/training/converters/"
    click Trainer "https://github.com/public-repositories/spacy/tree/master/spacy/training/"
    click Gatsby "https://github.com/public-repositories/spacy/tree/master/website/"
    click CI "https://github.com/public-repositories/spacy/tree/master/.github/workflows/"
    click CI "https://github.com/public-repositories/spacy/blob/master/azure-pipelines.yml"
    click Scripts "https://github.com/public-repositories/spacy/tree/master/bin/"
    click Tests "https://github.com/public-repositories/spacy/tree/master/spacy/tests/"

    %% Styles
    classDef python fill:#D6EAF8,stroke:#1B4F72,color:#1B2631
    classDef cython fill:#AED6F1,stroke:#21618C,color:#1B2631
    classDef external fill:#ABEBC6,stroke:#1D8348,color:#145A32
    classDef docs fill:#FAD7A0,stroke:#B9770E,color:#784212
    classDef ci fill:#E8DAEF,stroke:#7D3C98,color:#512E5F
    classDef storage fill:#F5CBA7,stroke:#B9770E,color:#784212
```
