How abchapman93/va_covid-19_nlp_bsv works
Project type and purpose
• This is an open-source NLP pipeline/library (Python package) designed to process clinical text for COVID-19 surveillance in VA data.
• It wraps a spaCy/medSpaCy processing pipeline, applies rule-based knowledge-base modules, and outputs per-document COVID-19 classifications and entity annotations.
File structure
• Top level:
• cov_bsv/ ← the Python package
• notebooks/ ← tutorials demonstrating pipeline usage
• tests/ ← unit tests
• setup.py, README, LICENSE
• cov_bsv/ contains:
• __init__.py ← public API, load() function
• document_classifier.py ← component assigning doc._.cov_classification
• util.py ← helper functions (e.g., load rules, visualization)
• knowledge_base/ ← rule definitions organized by phase:
– preprocess_rules.py
– target_rules.py
– concept_tag_rules.py
– section_rules.py
– context_rules.py
– postprocess_rules.py
– callbacks.py ← custom medSpaCy callbacks
• Dependencies: spaCy, medSpaCy; external model en_core_web_sm.
Insights from README
• The pipeline steps mirror the paper: load spaCy model, register medSpaCy components, load cov_bsv knowledge base, run through the chain of rule stages, then document classification.
• Notebooks illustrate quickstart, rule customization, classification, error correction.
a. Main Components
• Input Text (clinical note)
• cov_bsv.load() (API entry point)
• spaCy nlp object
• medSpaCy core components
• cov_bsv custom pipeline stages (preprocess, target matcher, concept tagger, section splitter, context, postprocess)
• Document Classifier component
• Output: Entities + doc._.cov_classification
• Knowledge Base (rules modules) loaded into each stage
b. Relationships & Data Flow
1. User script or notebook calls cov_bsv.load()
2. cov_bsv.load → spacy.load(en_core_web_sm)
3. Pipeline registration: medSpaCy components & cov_bsv callbacks
4. Each rule module in knowledge_base is injected into its pipeline component
5. Input text → nlp(text) → tokens & dependency parse
6. Sequential execution of pipeline stages (preprocess → target → concept → section → context → postprocess)
7. document_classifier inspects annotated doc → sets doc._.cov_classification
8. Output: annotated spaCy Doc + classification
c. Architectural patterns/design principles
• Pipeline pattern (linear, ordered processing stages)
• Separation of concerns: rule definitions vs. processing logic vs. classification
• Extensibility: drop-in rule modules, medSpaCy callbacks
d. Key technologies & libraries
• Python 3.6+
• spaCy (en_core_web_sm)
• medSpaCy
• cov_bsv (custom code)
• Jupyter/Colab notebooks for examples