How allenai/scispacy works
Main Components
a) Core Components:
• Custom Tokenizer Module
• POS Tagger & Syntactic Parser
• Entity Span Detection
• Abbreviation Detector
• Entity Linker
• Hyponym Detector
• Knowledge Base Integration
b) External Dependencies:
• spaCy Core
• UMLS Knowledge Base
• Other Knowledge Bases (MeSH, RxNorm, Gene Ontology, HPO)
Custom Pipeline Components:
• Custom Tokenizer
• POS Tagger
• Syntactic Parser
• Entity Recognition
• Entity Linking
Key Relationships to Highlight
a) Data Flow:
• Input Text → Custom Tokenizer → POS/Parser → Entity Detection → Entity Linking
• Abbreviation Detection can interact with Entity Linking
• Knowledge Base queries flow from Entity Linker to KB Storage
b) Integration Points:
• spaCy Pipeline Integration
• Knowledge Base Connectivity
• Model Loading/Saving
Specific Implementation Details
• Rectangles: Core processing components
• Hexagons: External interfaces
• Diamonds: Decision points
• Blue: Core processing components
• Green: Data storage
• Orange: External interfaces
• Purple: Knowledge bases
• Gray: Optional components
Technical Notes
Annotations for:
• Python 3.6+ requirement
• GPU support for SciBERT model
• Memory requirements for different models
• Knowledge base sizes (e.g., UMLS ~3M concepts)
Component Interactions
Detail the following workflows:
• Model loading and initialization
• Document processing pipeline
• Entity linking process
• Knowledge base querying
• Abbreviation detection and resolution
Remember to include:
• Version information
• System requirements
• Pipeline configuration options
• Data flow directions
• Optional component paths
This will provide a comprehensive view of scispaCy's architecture while maintaining clarity about how the components interact and can be configured.