# allenai/scispacy architecture

> Architecture diagram of the GitHub repository allenai/scispacy, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-02-03). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/allenai/scispacy
- Repository: https://github.com/allenai/scispacy

## Overview

##### Main Components

a) Core Components:
- Custom Tokenizer Module
- POS Tagger & Syntactic Parser
- Entity Span Detection
- Abbreviation Detector
- Entity Linker
- Hyponym Detector
- Knowledge Base Integration

b) External Dependencies:
- spaCy Core
- UMLS Knowledge Base
- Other Knowledge Bases (MeSH, RxNorm, Gene Ontology, HPO)

**Custom Pipeline Components:**
- Custom Tokenizer
- POS Tagger
- Syntactic Parser
- Entity Recognition
- Entity Linking

##### Key Relationships to Highlight

a) Data Flow:
- Input Text → Custom Tokenizer → POS/Parser → Entity Detection → Entity Linking
- Abbreviation Detection can interact with Entity Linking
- Knowledge Base queries flow from Entity Linker to KB Storage

b) Integration Points:
- spaCy Pipeline Integration
- Knowledge Base Connectivity
- Model Loading/Saving

##### Specific Implementation Details

- Rectangles: Core processing components
- Hexagons: External interfaces
- Diamonds: Decision points

- Blue: Core processing components
- Green: Data storage
- Orange: External interfaces
- Purple: Knowledge bases
- Gray: Optional components

##### Technical Notes

Annotations for:
- Python 3.6+ requirement
- GPU support for SciBERT model
- Memory requirements for different models
- Knowledge base sizes (e.g., UMLS ~3M concepts)

##### Component Interactions

Detail the following workflows:
- Model loading and initialization
- Document processing pipeline
- Entity linking process
- Knowledge base querying
- Abbreviation detection and resolution

Remember to include:
- Version information
- System requirements
- Pipeline configuration options
- Data flow directions
- Optional component paths

This will provide a comprehensive view of scispaCy's architecture while maintaining clarity about how the components interact and can be configured.

## Diagram (Mermaid)

```mermaid
graph TB
    subgraph Interface
        API[Python API Interface]
        ModelLoad[Model Loading Interface]:::interface
    end

    subgraph "Core Processing Pipeline"
        direction TB
        CustomTok[Custom Tokenizer]:::core
        POSParser[POS Tagger & Parser]:::core
        EntityDet[Entity Recognition]:::core
        AbbrevDet[Abbreviation Detector]:::core
        EntityLink[Entity Linker]:::core
        HypoDet[Hyponym Detector]:::core
        SentSeg[Sentence Segmenter]:::core
    end

    subgraph "Knowledge & Storage"
        direction TB
        KB[(Knowledge Bases)]:::storage
        Cache[(File Cache)]:::storage
        ModelStore[(Model Storage)]:::storage
        CandGen[Candidate Generator]:::core
    end

    subgraph "External Dependencies"
        SpaCy[spaCy Core]:::external
        UMLS[(UMLS KB)]:::external
        OtherKB[(Other KBs)]:::external
    end

    %% Relationships
    API --> ModelLoad
    ModelLoad --> CustomTok
    CustomTok --> POSParser
    POSParser --> EntityDet
    EntityDet --> AbbrevDet
    AbbrevDet --> EntityLink
    EntityLink --> HypoDet
    CustomTok --> SentSeg

    EntityLink --> CandGen
    CandGen --> KB
    KB --> UMLS
    KB --> OtherKB
    ModelLoad --> ModelStore
    Cache --> KB

    SpaCy --> CustomTok
    SpaCy --> POSParser
    SpaCy --> EntityDet

    %% Click Events
    click CustomTok "https://github.com/allenai/scispacy/blob/main/scispacy/custom_tokenizer.py"
    click POSParser "https://github.com/allenai/scispacy/blob/main/configs/base_parser_tagger.cfg"
    click EntityDet "https://github.com/allenai/scispacy/blob/main/configs/base_ner.cfg"
    click AbbrevDet "https://github.com/allenai/scispacy/blob/main/scispacy/abbreviation.py"
    click EntityLink "https://github.com/allenai/scispacy/blob/main/scispacy/linking.py"
    click HypoDet "https://github.com/allenai/scispacy/blob/main/scispacy/hyponym_detector.py"
    click KB "https://github.com/allenai/scispacy/blob/main/scispacy/umls_utils.py"
    click CandGen "https://github.com/allenai/scispacy/blob/main/scispacy/candidate_generation.py"
    click ModelStore "https://github.com/allenai/scispacy/blob/main/data/meta_*.json"
    click SentSeg "https://github.com/allenai/scispacy/blob/main/scispacy/custom_sentence_segmenter.py"
    click Cache "https://github.com/allenai/scispacy/blob/main/scispacy/file_cache.py"
    click UMLS "https://github.com/allenai/scispacy/tree/main/tests/fixtures/umls_META/"

    %% Styles
    classDef core fill:#3498db,stroke:#2980b9,color:white
    classDef storage fill:#2ecc71,stroke:#27ae60,color:white
    classDef external fill:#e74c3c,stroke:#c0392b,color:white
    classDef interface fill:#f39c12,stroke:#d35400,color:white

    %% Legend
    subgraph Legend
        L1[Core Components]:::core
        L2[(Storage Systems)]:::storage
        L3[External Systems]:::external
        L4[Interfaces]:::interface
    end
```
