How patrickthealba/hereditary-cancer-testing-nlp works
Project Type and Purpose
• This is a UIMA‐based NLP toolkit implemented in Java/Groovy that processes clinical notes to extract hereditary cancer information.
• It provides two pipelines:
a. Biomarker Pipeline – extracts biomarker-related entities and relationships.
b. Gene Mapping Pipeline – performs gene NER and mapping.
High-Level Architecture Components
• Data Ingestion Layer
• Readers (FileCollectionReader, DatabaseCollectionReader) configured via Groovy under config/readers/.
• NLP Processing Layer (UIMA Pipeline)
• BasePipeline – bootstraps UIMA, loads the TypeSystem.xml from generated-types/resources.
• Biomarker_Pipeline – sequence of Analysis Engines:
– RegexBasedSectionizer (section segmentation)
– BiomarkerLogic (pattern‐based entity extraction)
– VariantMappingLogic (if mapping variants)
– Other AE modules as defined under src/main/java/gov/va/vinci/ae/
• Type System – auto‐generated Java/Groovy classes under generated-types/ defining all annotations.
• Output Layer
• Listeners (BasicCsvListener, BasicDatabaseListener, etc.) under src/main/java/gov/va/vinci/listeners/ write annotations to CSV, DB, Chex, or binary formats.
• Configuration
• Groovy configs in config/ (ClientConfig.groovy, ServiceConfig.groovy, listener and reader configs).
• External Systems
• Database for storing results
• File system for input/output files
• (Optional) Web service endpoints via LeoClient/LeoService
Data Flow and Component Interaction
a. RegexBasedSectionizer → segments clinical note into sections
b. BiomarkerLogic → applies regex patterns from resources/*.pattern and resources/*.groovy to identify biomarker mentions
c. (Optionally) VariantMappingLogic → uses variant_mapping.pattern and associated anchor terms to normalize variant mentions
Key Architectural Patterns and Principles
• UIMA Pipeline Pattern: modular, pluggable Analysis Engines
• Configuration‐over‐code via Groovy scripts for readers, listeners, resource lists, and logging
• Code Generation for Type System ensures strong typing and consistency across pipelines
• Separation of concerns:
• Data ingestion vs. annotation logic vs. output handling
• Groovy config vs. Java pipeline code
Steps in the Biomarker Pipeline (to include
1. Load TypeSystem.xml and resource dictionaries (.pattern, .groovy)
2. Initialize FileCollectionReader with input file list
3. Create CAS and populate document text
4. Apply RegexBasedSectionizer AE → produce Section annotations
5. Apply BiomarkerLogic AE → produce Biomarker_Anchor, Biomarker_Pattern, Concept annotations
6. (Optional) Apply VariantMappingLogic AE → produce Variant_Mapping annotations
7. Route final CAS to configured Listener(s)
8. Listener writes output (CSV rows or DB insertions)