# cogstack/medcat architecture

> Architecture diagram of the GitHub repository cogstack/medcat, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-02). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/cogstack/medcat
- Repository: https://github.com/cogstack/medcat

## Overview

##### Project Type and Purpose
- Open-source Python library for clinical Natural Language Processing (NLP)
- Core functionality: extract concepts from free text and link them to ontologies (e.g., SNOMED-CT, UMLS)
- Provides both vocabulary-based and transformer-based NER, context-based linking, and meta‐classification

##### File Structure to Derive Components
- medcat/
– config*.py: loading & managing CDB/model settings
– cdb_maker.py: building Concept DBs from ontology source data
– cat.py: main API entrypoint (MedCAT.Cat class)
– datasets/: data loaders & collators for self-supervised and supervised training
– preprocessing/: tokenizers, cleaners, taggers
– ner/: vocabulary & transformer NER modules
– linking/: context-based disambiguation classes
– meta_cat.py: meta-classification (negation, experiencer)
– pipeline/: orchestration runner that ties NER, linking, meta steps
– utils/: helpers (config utils, saving/serialization, logging, metrics)
- configs/: example config files for training/validation
- examples/, notebooks/: demo usage & tutorials
- tests/: comprehensive unit/integration tests
- docs/: documentation templates & build scripts

##### Read README for Additional Insights
- MedCAT is library-first: no built-in server or UI
- Depends on spaCy, PyTorch/Transformers, jsonpickle
- Supports GPU/CPU modes
- Extensible: users plug in custom tokenizers, linkers, meta components

##### Main Components & Their Interactions
a. Input Layer
– External text sources (EHRs, clinical notes)
– Ontology packs (model downloads: CDB files for UMLS/SNOMED)
b. Preprocessing
– Tokenization (spaCy/transformers)
– Text cleaning, tagging (lowercasing, punctuation removal)
c. Named-Entity Recognition (NER)
– Vocabulary-based NER (gazetteer lookup)
– Transformer-based NER (Hugging Face models)
d. Entity Linking
– Vector-based context linker
– Rule-based disambiguation (e.g., highest similarity)
e. MetaCAT (Meta-classification)
– Negation, experiencer, hypothetical classification
f. Pipeline Runner
– medcat.pipeline.pipe_runner: chains preprocessing → NER → linking → meta
– Batch processing & multiprocessing for large corpora
g. Persistence and Utilities
– Saving/loading CDBs & checkpoints (jsonpickle serializer)
– Configuration management (config_utils)
– Logging hooks (console & file via medcat.add_default_log_handlers)
h. Training & Evaluation
– Self-supervised training (datasets/data_collator)
– Supervised fine-tuning (transformer pipelines)
– Regression & k-fold statistics modules (medcat.stats)

##### Architectural Patterns & Design Principles
- Modular pipeline: each stage (preprocess, ner, link, meta) is a reusable component
- Factory/config-driven: instantiate models/linkers via JSON/YAML (configs/)
- Plugin-friendly: users can subclass tokenizers, linkers, regressors
- Separation of concerns: medcat.utils holds infra code, medcat.pipeline orchestrates

##### Final Checklist
– Side note for configuration: point to configs/*.txt and pyproject.toml/requirements.txt

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% External Inputs
    A[/"External Text Sources (EHRs, Clinical Notes)"/]:::external
    B[/"Ontology Downloads (UMLS/SNOMED)"/]:::external
    C[/"Transformer Checkpoints (Hugging Face)"/]:::external

    %% Pipeline Stages
    subgraph "Preprocessing"
        direction TB
        P1["Tokenizer\n(medcat.preprocessing.tokenizers)"]:::pre
        P2["Cleaner\n(medcat.preprocessing.cleaners)"]:::pre
        P3["Tagger\n(medcat.preprocessing.taggers)"]:::pre
    end

    subgraph "Named-Entity Recognition"
        direction TB
        N1["VocabNER\n(medcat.ner.vocab_based_ner)"]:::ner
        N2["TransformerNER\n(medcat.ner.transformers_ner)"]:::ner
    end

    subgraph "Entity Linking"
        direction TB
        L1["ContextBasedLinker\n(medcat.linking.context_based_linker)"]:::link
        L2["VectorContextModel\n(medcat.linking.vector_context_model)"]:::link
    end

    subgraph "MetaCAT"
        direction TB
        M1["MetaClassifier\n(medcat.meta_cat)"]:::meta
    end

    subgraph "Pipeline Runner"
        direction TB
        R1["PipeRunner\n(medcat.pipeline.pipe_runner)"]:::runner
        R2["Entrypoint Alias\n(medcat/pipe.py)"]:::runner
    end

    subgraph "CDB Management"
        direction TB
        CDB1["CDB Maker\n(medcat.cdb_maker)"]:::db
        CDB2["CDB API\n(medcat.cdb)"]:::db
    end

    subgraph "Configuration Utilities"
        CU["Config Utils\n(medcat/utils/config_utils)"]:::util
    end

    subgraph "Serialization & Persistence"
        SP["Serializer\n(medcat/utils/saving/serializer)"]:::util
    end

    subgraph "Utilities & Helpers"
        direction TB
        U1["Utils Core"]:::util
    end

    subgraph "Training & Data Loading"
        direction TB
        D1["Datasets & Collators\n(medcat/datasets)"]:::data
    end

    subgraph "Statistics & Evaluation"
        direction TB
        S1["Stats & K-Fold\n(medcat/stats)"]:::data
    end

    API["MedCAT.Cat\n(medcat/cat.py)"]:::api
    Configs["Example Configs\n(configs/)"]:::config

    %% Data Flow
    A --> P1
    P1 --> P2 --> P3 --> N1
    P3 --> N2
    N1 --> L1 --> L2 --> M1 --> API
    N2 --> L1
    B --> CDB1 --> CDB2 --> API
    C --> N2
    API --> R1 --> SP
    R1 --> CU
    R1 --> U1
    R1 --> D1 --> S1
    Configs --> CU
    API --> Configs

    %% Click Events
    click P1 "https://github.com/cogstack/medcat/tree/master/medcat/preprocessing"
    click P2 "https://github.com/cogstack/medcat/tree/master/medcat/preprocessing"
    click P3 "https://github.com/cogstack/medcat/tree/master/medcat/preprocessing"
    click N1 "https://github.com/cogstack/medcat/tree/master/medcat/ner"
    click N2 "https://github.com/cogstack/medcat/tree/master/medcat/ner"
    click L1 "https://github.com/cogstack/medcat/tree/master/medcat/linking"
    click L2 "https://github.com/cogstack/medcat/tree/master/medcat/linking"
    click M1 "https://github.com/cogstack/medcat/blob/master/medcat/meta_cat.py"
    click CU "https://github.com/cogstack/medcat/blob/master/medcat/utils/config_utils.py"
    click SP "https://github.com/cogstack/medcat/blob/master/medcat/utils/saving/serializer.py"
    click R1 "https://github.com/cogstack/medcat/blob/master/medcat/pipeline/pipe_runner.py"
    click API "https://github.com/cogstack/medcat/blob/master/medcat/cat.py"
    click CDB1 "https://github.com/cogstack/medcat/blob/master/medcat/cdb_maker.py"
    click CDB2 "https://github.com/cogstack/medcat/blob/master/medcat/cdb.py"
    click Configs "https://github.com/cogstack/medcat/tree/master/configs"
    click D1 "https://github.com/cogstack/medcat/tree/master/medcat/datasets"
    click S1 "https://github.com/cogstack/medcat/tree/master/medcat/stats"

    %% Styles
    classDef external fill:#f2f2f2,stroke:#999,stroke-width:1px
    classDef pre fill:#cce5ff,stroke:#3399ff,stroke-width:2px
    classDef ner fill:#d4edda,stroke:#28a745,stroke-width:2px
    classDef link fill:#ffe5cc,stroke:#ff9933,stroke-width:2px
    classDef meta fill:#e5ccff,stroke:#9933ff,stroke-width:2px
    classDef runner fill:#fff3cd,stroke:#ffcc00,stroke-width:2px
    classDef db fill:#e2e3e5,stroke:#6c757d,stroke-width:2px
    classDef util fill:#f8f9fa,stroke:#adb5bd,stroke-width:1px
    classDef data fill:#e9ecef,stroke:#6c757d,stroke-width:1px
    classDef api fill:#f5f5f5,stroke:#6c757d,stroke-width:2px
    classDef config fill:#ffffff,stroke:#adb5bd,stroke-width:1px,stroke-dasharray: 5 2
```
