How cogstack/medcat works
Project Type and Purpose
• Open-source Python library for clinical Natural Language Processing (NLP)
• Core functionality: extract concepts from free text and link them to ontologies (e.g., SNOMED-CT, UMLS)
• Provides both vocabulary-based and transformer-based NER, context-based linking, and meta‐classification
File Structure to Derive Components
• medcat/
– config*.py: loading & managing CDB/model settings
– cdb_maker.py: building Concept DBs from ontology source data
– cat.py: main API entrypoint (MedCAT.Cat class)
– datasets/: data loaders & collators for self-supervised and supervised training
– preprocessing/: tokenizers, cleaners, taggers
– ner/: vocabulary & transformer NER modules
– linking/: context-based disambiguation classes
– meta_cat.py: meta-classification (negation, experiencer)
– pipeline/: orchestration runner that ties NER, linking, meta steps
– utils/: helpers (config utils, saving/serialization, logging, metrics)
• configs/: example config files for training/validation
• examples/, notebooks/: demo usage & tutorials
• tests/: comprehensive unit/integration tests
• docs/: documentation templates & build scripts
Read README for Additional Insights
• MedCAT is library-first: no built-in server or UI
• Depends on spaCy, PyTorch/Transformers, jsonpickle
• Supports GPU/CPU modes
• Extensible: users plug in custom tokenizers, linkers, meta components
Main Components & Their Interactions
a. Input Layer
– External text sources (EHRs, clinical notes)
– Ontology packs (model downloads: CDB files for UMLS/SNOMED)
b. Preprocessing
– Tokenization (spaCy/transformers)
– Text cleaning, tagging (lowercasing, punctuation removal)
c. Named-Entity Recognition (NER)
– Vocabulary-based NER (gazetteer lookup)
– Transformer-based NER (Hugging Face models)
d. Entity Linking
– Vector-based context linker
– Rule-based disambiguation (e.g., highest similarity)
e. MetaCAT (Meta-classification)
– Negation, experiencer, hypothetical classification
f. Pipeline Runner
– medcat.pipeline.pipe_runner: chains preprocessing → NER → linking → meta
– Batch processing & multiprocessing for large corpora
g. Persistence and Utilities
– Saving/loading CDBs & checkpoints (jsonpickle serializer)
– Configuration management (config_utils)
– Logging hooks (console & file via medcat.add_default_log_handlers)
h. Training & Evaluation
– Self-supervised training (datasets/data_collator)
– Supervised fine-tuning (transformer pipelines)
– Regression & k-fold statistics modules (medcat.stats)
Architectural Patterns & Design Principles
• Modular pipeline: each stage (preprocess, ner, link, meta) is a reusable component
• Factory/config-driven: instantiate models/linkers via JSON/YAML (configs/)
• Plugin-friendly: users can subclass tokenizers, linkers, regressors
• Separation of concerns: medcat.utils holds infra code, medcat.pipeline orchestrates
Final Checklist
– Side note for configuration: point to configs/*.txt and pyproject.toml/requirements.txt