# abchapman93/va_covid-19_nlp_bsv architecture

> Architecture diagram of the GitHub repository abchapman93/va_covid-19_nlp_bsv, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-12). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/abchapman93/va_covid-19_nlp_bsv
- Repository: https://github.com/abchapman93/va_covid-19_nlp_bsv

## Overview

##### Project type and purpose
- This is an open-source NLP pipeline/library (Python package) designed to process clinical text for COVID-19 surveillance in VA data.
- It wraps a spaCy/medSpaCy processing pipeline, applies rule-based knowledge-base modules, and outputs per-document COVID-19 classifications and entity annotations.

##### File structure
- Top level:
- cov_bsv/         ← the Python package
- notebooks/       ← tutorials demonstrating pipeline usage
- tests/           ← unit tests
- setup.py, README, LICENSE
- cov_bsv/ contains:
- __init__.py      ← public API, load() function
- document_classifier.py  ← component assigning doc._.cov_classification
- util.py          ← helper functions (e.g., load rules, visualization)
- knowledge_base/  ← rule definitions organized by phase:
– preprocess_rules.py
– target_rules.py
– concept_tag_rules.py
– section_rules.py
– context_rules.py
– postprocess_rules.py
– callbacks.py    ← custom medSpaCy callbacks
- Dependencies: spaCy, medSpaCy; external model en_core_web_sm.

##### Insights from README
- The pipeline steps mirror the paper: load spaCy model, register medSpaCy components, load cov_bsv knowledge base, run through the chain of rule stages, then document classification.
- Notebooks illustrate quickstart, rule customization, classification, error correction.

a. Main Components
- Input Text (clinical note)
- cov_bsv.load() (API entry point)
- spaCy nlp object
- medSpaCy core components
- cov_bsv custom pipeline stages (preprocess, target matcher, concept tagger, section splitter, context, postprocess)
- Document Classifier component
- Output: Entities + doc._.cov_classification
- Knowledge Base (rules modules) loaded into each stage
b. Relationships & Data Flow
1. User script or notebook calls cov_bsv.load()
2. cov_bsv.load → spacy.load(en_core_web_sm)
3. Pipeline registration: medSpaCy components & cov_bsv callbacks
4. Each rule module in knowledge_base is injected into its pipeline component
5. Input text → nlp(text) → tokens & dependency parse
6. Sequential execution of pipeline stages (preprocess → target → concept → section → context → postprocess)
7. document_classifier inspects annotated doc → sets doc._.cov_classification
8. Output: annotated spaCy Doc + classification
c. Architectural patterns/design principles
- Pipeline pattern (linear, ordered processing stages)
- Separation of concerns: rule definitions vs. processing logic vs. classification
- Extensibility: drop-in rule modules, medSpaCy callbacks
d. Key technologies & libraries
- Python 3.6+
- spaCy (en_core_web_sm)
- medSpaCy
- cov_bsv (custom code)
- Jupyter/Colab notebooks for examples

## Diagram (Mermaid)

```mermaid
flowchart TB
  %% Initialization
  subgraph "Initialization"
    API["cov_bsv.load()"]:::api
    SPACY["spaCy.load(en_core_web_sm)"]:::deps
    MEDSPACY["medSpaCy Components"]:::deps
    API --> SPACY
    API --> MEDSPACY
  end

  %% Rule Injection
  subgraph "Rule Injection"
    KB_PRE["Preprocess Rules"]:::kb
    KB_TAR["Target Matching Rules"]:::kb
    KB_CONCEPT["Concept Tagging Rules"]:::kb
    KB_SECTION["Section Splitting Rules"]:::kb
    KB_CONTEXT["Context Rules"]:::kb
    KB_POST["Postprocess Rules"]:::kb
    CALLBACKS["MedSpaCy Callbacks"]:::kb
    MEDSPACY --> CALLBACKS
    API --> KB_PRE
    API --> KB_TAR
    API --> KB_CONCEPT
    API --> KB_SECTION
    API --> KB_CONTEXT
    API --> KB_POST
  end

  %% Pipeline Execution
  subgraph "Pipeline Execution"
    INPUT["Input: Clinical Text"]:::data
    DOC["spaCy Doc Object"]:::data
    PREPROCESS["Preprocess Stage"]:::stage
    TARGET["Target Matching Stage"]:::stage
    CONCEPT["Concept Tagging Stage"]:::stage
    SECTION["Section Splitting Stage"]:::stage
    CONTEXT["Context Analysis Stage"]:::stage
    POSTPROCESS["Postprocess Stage"]:::stage
    CLASSIFY["Document Classifier"]:::stage
    OUTPUT["Output: Annotated Doc + Classification"]:::data

    INPUT --> DOC
    DOC --> PREPROCESS
    PREPROCESS --> TARGET
    TARGET --> CONCEPT
    CONCEPT --> SECTION
    SECTION --> CONTEXT
    CONTEXT --> POSTPROCESS
    POSTPROCESS --> CLASSIFY
    CLASSIFY --> OUTPUT
  end

  %% Extensions & Artifacts
  subgraph "Extensions & Artifacts"
    UTIL["Utility Functions"]:::lib
    NB0["00-quickstart.ipynb"]:::notebook
    NB1["01-target-matching.ipynb"]:::notebook
    NB2["02-attribute-assertion.ipynb"]:::notebook
    NB3["03-document-classification.ipynb"]:::notebook
    NB4["04-fixing-errors.ipynb"]:::notebook
    TESTS["Unit Tests"]:::test
    IMAGES["Static Assets & Diagrams"]:::asset
    SETUP["setup.py"]:::asset

    API --> UTIL
    OUTPUT --> UTIL
    API --> NB0
    API --> NB1
    API --> NB2
    API --> NB3
    API --> NB4
    API --> TESTS
    API --> IMAGES
    API --> SETUP
  end

  %% Click Events
  click API "https://github.com/abchapman93/va_covid-19_nlp_bsv/blob/master/cov_bsv/__init__.py"
  click CLASSIFY "https://github.com/abchapman93/va_covid-19_nlp_bsv/blob/master/cov_bsv/document_classifier.py"
  click UTIL "https://github.com/abchapman93/va_covid-19_nlp_bsv/blob/master/cov_bsv/util.py"
  click KB_PRE "https://github.com/abchapman93/va_covid-19_nlp_bsv/blob/master/cov_bsv/knowledge_base/preprocess_rules.py"
  click KB_TAR "https://github.com/abchapman93/va_covid-19_nlp_bsv/blob/master/cov_bsv/knowledge_base/target_rules.py"
  click KB_CONCEPT "https://github.com/abchapman93/va_covid-19_nlp_bsv/blob/master/cov_bsv/knowledge_base/concept_tag_rules.py"
  click KB_SECTION "https://github.com/abchapman93/va_covid-19_nlp_bsv/blob/master/cov_bsv/knowledge_base/section_rules.py"
  click KB_CONTEXT "https://github.com/abchapman93/va_covid-19_nlp_bsv/blob/master/cov_bsv/knowledge_base/context_rules.py"
  click KB_POST "https://github.com/abchapman93/va_covid-19_nlp_bsv/blob/master/cov_bsv/knowledge_base/postprocess_rules.py"
  click CALLBACKS "https://github.com/abchapman93/va_covid-19_nlp_bsv/blob/master/cov_bsv/knowledge_base/callbacks.py"
  click NB0 "https://github.com/abchapman93/va_covid-19_nlp_bsv/blob/master/notebooks/00-quickstart.ipynb"
  click NB1 "https://github.com/abchapman93/va_covid-19_nlp_bsv/blob/master/notebooks/01-target-matching.ipynb"
  click NB2 "https://github.com/abchapman93/va_covid-19_nlp_bsv/blob/master/notebooks/02-attribute-assertion.ipynb"
  click NB3 "https://github.com/abchapman93/va_covid-19_nlp_bsv/blob/master/notebooks/03-document-classification.ipynb"
  click NB4 "https://github.com/abchapman93/va_covid-19_nlp_bsv/blob/master/notebooks/04-fixing-errors.ipynb"
  click TESTS "https://github.com/abchapman93/va_covid-19_nlp_bsv/blob/master/tests/test_cov_bsv.py"
  click SETUP "https://github.com/abchapman93/va_covid-19_nlp_bsv/blob/master/setup.py"
  click IMAGES "https://github.com/abchapman93/va_covid-19_nlp_bsv/tree/master/images/"

  %% Styles
  classDef api fill:#D6EAF8,stroke:#2980B9,color:#1B4F72;
  classDef deps fill:#F2F3F4,stroke:#A6ACAF,color:#424949;
  classDef kb fill:#FCF3CF,stroke:#F1C40F,color:#7D6608;
  classDef stage fill:#D5F5E3,stroke:#28B463,color:#1E8449;
  classDef data fill:#FADBD8,stroke:#E74C3C,color:#78281F;
  classDef lib fill:#F5EEF8,stroke:#A569BD,color:#4A235A;
  classDef notebook fill:#EBF5FB,stroke:#5DADE2,color:#1B4F72;
  classDef test fill:#FDEDEC,stroke:#CD6155,color:#922B21;
  classDef asset fill:#ECECEC,stroke:#808B96,color:#2C3E50;
```
