# emilyalsentzer/clinicalbert architecture

> Architecture diagram of the GitHub repository emilyalsentzer/clinicalbert, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-31). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/emilyalsentzer/clinicalbert
- Repository: https://github.com/emilyalsentzer/clinicalbert

## Overview

This project, clinicalBERT, is a research and experimentation repository focused on pretraining and fine-tuning deep language models (specifically BERT-derived models) on clinical data from sources such as MIMIC. It provides components for both language model pretraining (including formatting and fine-tuning on clinical data) and for downstream tasks such as natural language inference (NLI) and named entity recognition (NER) evaluation.

##### Main Components
a. Pretraining Pipeline:
- Data Formatting: Represent the module responsible for converting raw clinical datasets into input suitable for BERT. This can be shown as a block labeled “MIMIC Data Formatter” (source: format_mimic_for_BERT.py).
- Pretraining Data Creation: A block for generating pretraining datasets using create_pretrain_data.sh.
- Language Model Pretraining: A block representing the model fine-tuning process using scripts like finetune_lm_tf.sh and run_pretraining.py.
- Tokenization & Heuristics: Include a node for heuristic_tokenize.py which indicates data preprocessing steps.
b. Downstream Tasks:
- i2b2 Preprocessing for NER: Several blocks representing different i2b2 components (2006 de-identification, 2010 relations, etc.). Each may be noted separately if detail is needed, or grouped under a “i2b2 Data Preprocessing” block.
- NER Evaluation: A block that includes components like conlleval.pl, format_for_i2b2_eval.py, ner_detokenize.py, and score_i2b2.py.
- Classification Example: A block for the Med NLI task (using run_classifier.py and run_classifier.sh) as illustrated in the README.
c. External Dependencies/Models:
- Pretrained BERT/BioBERT: A clear indication that the project builds upon the BioBERT models and the cased BERT model (cased_L-12_H-768_A-12). These external models should be shown as separate components feeding into the pretraining process.
- Clinical Data Sources: Represent external clinical datasets such as MIMIC and i2b2.

##### Relationships and Data Flow
- Flow from “MIMIC Data Formatter” to “Pretraining Data Creation” and then to the “Language Model Pretraining” block.
- For downstream tasks, indicate how preprocessed i2b2 data moves into evaluation and classifier components.
- That the output of the pretraining module (Clinical BERT models) is used across both the downstream classification and NER tasks.

##### Architectural Patterns and Design Principles
- Separation of Concerns: The repository is cleanly divided into two main directories (lm_pretraining and downstream_tasks) that represent distinct workflows.
- Pipeline Architecture: Emphasize the sequential pipeline for model pretraining, from raw data to formatted data to fine-tuned models.
- Reusability and Modularity: Downstream tasks act as example applications showing how the pre-trained models can be integrated into various clinical NLP tasks.

- External libraries or frameworks if needed—most notably the usage of Hugging Face’s Transformers library for model deployment (as mentioned in the README).
- For completeness, add a note for environment setup such as dependencies coming from requirements.txt and any configuration inferred from the file structure.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% External Data Sources and Models
    MIMIC_DS["MIMIC Clinical Data"]:::external
    I2B2_DS["i2b2 Clinical Data"]:::external
    BioBERT["Pretrained BioBERT"]:::external
    BERTCAS["Pretrained BERT (cased)"]:::external

    %% Pretraining Pipeline Subgraph
    subgraph "Pretraining Pipeline"
        DF["MIMIC Data Formatter"]:::script
        HT["heuristic_tokenize.py"]:::script
        PD["Pretraining Data Creation"]:::script
        subgraph "Language Model Pretraining"
            L1["finetune_lm_tf.sh"]:::script
            L2["run_pretraining.py"]:::script
        end
        PM["Pretrained Clinical BERT Models"]:::model
    end

    %% Downstream Tasks Subgraph
    subgraph "Downstream Tasks"
        IP["i2b2 Data Preprocessing"]:::script
        NE["NER Evaluation"]:::script
        subgraph "Med NLI Task"
            C1["run_classifier.py"]:::script
            C2["run_classifier.sh"]:::script
        end
    end

    %% Pretraining Pipeline Flow
    MIMIC_DS -->|"ingests"| DF
    DF -->|"formats"| HT
    HT -->|"creates"| PD
    PD -->|"triggers"| L1
    L1 -->|"executes"| L2
    L2 -->|"outputs"| PM
    BioBERT -->|"provides"| PM
    BERTCAS -->|"provides"| PM

    %% Downstream Tasks Flow
    I2B2_DS -->|"ingests"| IP
    IP -->|"processes"| NE
    PM -->|"supplies"| NE
    PM -->|"supplies"| C1
    C1 -->|"orchestrates"| C2

    %% Feedback Loop from Evaluation to Pretraining
    NE -.->|"feedback"| L1

    %% Click Events
    click DF "https://github.com/emilyalsentzer/clinicalbert/blob/master/lm_pretraining/format_mimic_for_BERT.py"
    click PD "https://github.com/emilyalsentzer/clinicalbert/blob/master/lm_pretraining/create_pretrain_data.sh"
    click L1 "https://github.com/emilyalsentzer/clinicalbert/blob/master/lm_pretraining/finetune_lm_tf.sh"
    click L2 "https://github.com/emilyalsentzer/clinicalbert/blob/master/lm_pretraining/run_pretraining.py"
    click HT "https://github.com/emilyalsentzer/clinicalbert/blob/master/lm_pretraining/heuristic_tokenize.py"
    click IP "https://github.com/emilyalsentzer/clinicalbert/tree/master/downstream_tasks/i2b2_preprocessing"
    click NE "https://github.com/emilyalsentzer/clinicalbert/tree/master/downstream_tasks/ner_eval"
    click C1 "https://github.com/emilyalsentzer/clinicalbert/blob/master/downstream_tasks/run_classifier.py"
    click C2 "https://github.com/emilyalsentzer/clinicalbert/blob/master/downstream_tasks/run_classifier.sh"

    %% Class Definitions for Color Coding
    classDef script fill:#ADFF2F,stroke:#333,stroke-width:2px;
    classDef model fill:#87CEFA,stroke:#333,stroke-width:2px;
    classDef external fill:#FFA07A,stroke:#333,stroke-width:2px;
```
