# patrickthealba/hereditary-cancer-testing-nlp architecture

> Architecture diagram of the GitHub repository patrickthealba/hereditary-cancer-testing-nlp, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-12). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/patrickthealba/hereditary-cancer-testing-nlp
- Repository: https://github.com/patrickthealba/hereditary-cancer-testing-nlp

## Overview

##### Project Type and Purpose
- This is a UIMA‐based NLP toolkit implemented in Java/Groovy that processes clinical notes to extract hereditary cancer information.
- It provides two pipelines:
a. Biomarker Pipeline – extracts biomarker-related entities and relationships.
b. Gene Mapping Pipeline – performs gene NER and mapping.

##### High-Level Architecture Components
- Data Ingestion Layer
- Readers (FileCollectionReader, DatabaseCollectionReader) configured via Groovy under config/readers/.
- NLP Processing Layer (UIMA Pipeline)
- BasePipeline – bootstraps UIMA, loads the TypeSystem.xml from generated-types/resources.
- Biomarker_Pipeline – sequence of Analysis Engines:
– RegexBasedSectionizer (section segmentation)
– BiomarkerLogic (pattern‐based entity extraction)
– VariantMappingLogic (if mapping variants)
– Other AE modules as defined under src/main/java/gov/va/vinci/ae/
- Type System – auto‐generated Java/Groovy classes under generated-types/ defining all annotations.
- Output Layer
- Listeners (BasicCsvListener, BasicDatabaseListener, etc.) under src/main/java/gov/va/vinci/listeners/ write annotations to CSV, DB, Chex, or binary formats.
- Configuration
- Groovy configs in config/ (ClientConfig.groovy, ServiceConfig.groovy, listener and reader configs).
- External Systems
- Database for storing results
- File system for input/output files
- (Optional) Web service endpoints via LeoClient/LeoService

##### Data Flow and Component Interaction
a. RegexBasedSectionizer → segments clinical note into sections
b. BiomarkerLogic → applies regex patterns from resources/*.pattern and resources/*.groovy to identify biomarker mentions
c. (Optionally) VariantMappingLogic → uses variant_mapping.pattern and associated anchor terms to normalize variant mentions

##### Key Architectural Patterns and Principles
- UIMA Pipeline Pattern: modular, pluggable Analysis Engines
- Configuration‐over‐code via Groovy scripts for readers, listeners, resource lists, and logging
- Code Generation for Type System ensures strong typing and consistency across pipelines
- Separation of concerns:
- Data ingestion vs. annotation logic vs. output handling
- Groovy config vs. Java pipeline code

##### Steps in the Biomarker Pipeline (to include
1. Load TypeSystem.xml and resource dictionaries (.pattern, .groovy)
2. Initialize FileCollectionReader with input file list
3. Create CAS and populate document text
4. Apply RegexBasedSectionizer AE → produce Section annotations
5. Apply BiomarkerLogic AE → produce Biomarker_Anchor, Biomarker_Pattern, Concept annotations
6. (Optional) Apply VariantMappingLogic AE → produce Variant_Mapping annotations
7. Route final CAS to configured Listener(s)
8. Listener writes output (CSV rows or DB insertions)

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Shared Resources
    subgraph "Shared Resources"
        TS["TypeSystem.xml"]:::resource
        RD["Resource Dictionaries & Patterns"]:::resource
    end

    %% Data Ingestion Layer
    subgraph "Data Ingestion" 
        FCR["FileCollectionReader"]:::ingestion
        DBCR["DatabaseCollectionReader"]:::ingestion
        BBDBR["BatchDatabaseCollectionReader"]:::ingestion
    end

    %% UIMA Processing Layer
    subgraph "UIMA Pipeline"
        BP["BasePipeline"]:::pipeline
        subgraph "Biomarker Pipeline Steps"
            S1["1. Load TypeSystem & Resource Dictionaries"]:::pipeline
            S2["2. Initialize FileCollectionReader"]:::pipeline
            S3["3. Create CAS & Populate Text"]:::pipeline
            S4["4. RegexBasedSectionizer AE"]:::pipeline
            S5["5. BiomarkerLogic AE"]:::pipeline
            S6["6. VariantMappingLogic AE (optional)"]:::pipeline
            S7["7. Route CAS to Listener(s)"]:::pipeline
        end
    end

    %% Output Layer
    subgraph "Output Layer"
        CSV["BasicCsvListener"]:::output
        DBL["BasicDatabaseListener"]:::output
        BL["BinaryLearningListener"]:::output
        CL["ChexListener"]:::output
    end

    %% Web Services
    subgraph "External Systems"
        FC["File System"]:::external
        DB["Database"]:::external
        LC["LeoClient"]:::external
        LS["LeoService"]:::external
    end

    %% Connections
    FC -->|reads files| FCR
    DB -->|reads records| DBCR
    DB -->|reads batches| BBDBR
    FCR --> BP
    DBCR --> BP
    BBDBR --> BP
    BP --> S1
    S1 --> S2
    S2 --> S3
    S3 --> S4
    S4 --> S5
    S5 --> S6
    S6 --> S7
    S7 --> CSV
    S7 --> DBL
    S7 --> BL
    S7 --> CL
    LC -->|invoke pipeline| LS
    LS --> BP

    %% Click Events for Configs and Code
    click FCR "https://github.com/patrickthealba/hereditary-cancer-testing-nlp/blob/main/config/readers/FileCollectionReaderConfig.groovy"
    click DBCR "https://github.com/patrickthealba/hereditary-cancer-testing-nlp/blob/main/config/readers/DatabaseCollectionReader.groovy"
    click BBDBR "https://github.com/patrickthealba/hereditary-cancer-testing-nlp/blob/main/config/readers/BatchDatabaseCollectionReader.groovy"
    click TS "https://github.com/patrickthealba/hereditary-cancer-testing-nlp/blob/main/generated-types/resources/types/TypeSystem.xml"
    click RD "https://github.com/patrickthealba/hereditary-cancer-testing-nlp/tree/main/src/main/java/resources"
    click BP "https://github.com/patrickthealba/hereditary-cancer-testing-nlp/blob/main/src/main/java/gov/va/vinci/pipeline/BasePipeline.java"
    click S4 "https://github.com/patrickthealba/hereditary-cancer-testing-nlp/blob/main/src/main/java/gov/va/vinci/ae/RegexBasedSectionizer.java"
    click S5 "https://github.com/patrickthealba/hereditary-cancer-testing-nlp/blob/main/src/main/java/gov/va/vinci/ae/BiomarkerLogic.java"
    click S6 "https://github.com/patrickthealba/hereditary-cancer-testing-nlp/blob/main/src/main/java/gov/va/vinci/ae/VariantMappingLogic.java"
    click CSV "https://github.com/patrickthealba/hereditary-cancer-testing-nlp/blob/main/src/main/java/gov/va/vinci/listeners/BasicCsvListener.java"
    click DBL "https://github.com/patrickthealba/hereditary-cancer-testing-nlp/blob/main/src/main/java/gov/va/vinci/listeners/BasicDatabaseListener.java"
    click BL "https://github.com/patrickthealba/hereditary-cancer-testing-nlp/blob/main/src/main/java/gov/va/vinci/listeners/BinaryLearningListener.java"
    click CL "https://github.com/patrickthealba/hereditary-cancer-testing-nlp/blob/main/src/main/java/gov/va/vinci/listeners/ChexListener.java"
    click LC "https://github.com/patrickthealba/hereditary-cancer-testing-nlp/blob/main/src/main/java/gov/va/vinci/LeoClient.java"
    click LS "https://github.com/patrickthealba/hereditary-cancer-testing-nlp/blob/main/src/main/java/gov/va/vinci/LeoService.java"
    
    %% Styles
    classDef ingestion fill:#D0E7FF,stroke:#0366D6,color:#0366D6;
    classDef pipeline fill:#E4F7D2,stroke:#2E8B57,color:#2E8B57;
    classDef output fill:#FFE4B5,stroke:#FF8C00,color:#FF8C00;
    classDef resource fill:#E0E0E0,stroke:#808080,color:#404040;
    classDef external fill:#F0E0FF,stroke:#800080,color:#800080;
```
