How emilyalsentzer/clinicalbert works
This project, clinicalBERT, is a research and experimentation repository focused on pretraining and fine-tuning deep language models (specifically BERT-derived models) on clinical data from sources such as MIMIC. It provides components for both language model pretraining (including formatting and fine-tuning on clinical data) and for downstream tasks such as natural language inference (NLI) and named entity recognition (NER) evaluation.
Main Components
a. Pretraining Pipeline:
• Data Formatting: Represent the module responsible for converting raw clinical datasets into input suitable for BERT. This can be shown as a block labeled “MIMIC Data Formatter” (source: format_mimic_for_BERT.py).
• Pretraining Data Creation: A block for generating pretraining datasets using create_pretrain_data.sh.
• Language Model Pretraining: A block representing the model fine-tuning process using scripts like finetune_lm_tf.sh and run_pretraining.py.
• Tokenization & Heuristics: Include a node for heuristic_tokenize.py which indicates data preprocessing steps.
b. Downstream Tasks:
• i2b2 Preprocessing for NER: Several blocks representing different i2b2 components (2006 de-identification, 2010 relations, etc.). Each may be noted separately if detail is needed, or grouped under a “i2b2 Data Preprocessing” block.
• NER Evaluation: A block that includes components like conlleval.pl, format_for_i2b2_eval.py, ner_detokenize.py, and score_i2b2.py.
• Classification Example: A block for the Med NLI task (using run_classifier.py and run_classifier.sh) as illustrated in the README.
c. External Dependencies/Models:
• Pretrained BERT/BioBERT: A clear indication that the project builds upon the BioBERT models and the cased BERT model (cased_L-12_H-768_A-12). These external models should be shown as separate components feeding into the pretraining process.
• Clinical Data Sources: Represent external clinical datasets such as MIMIC and i2b2.
Relationships and Data Flow
• Flow from “MIMIC Data Formatter” to “Pretraining Data Creation” and then to the “Language Model Pretraining” block.
• For downstream tasks, indicate how preprocessed i2b2 data moves into evaluation and classifier components.
• That the output of the pretraining module (Clinical BERT models) is used across both the downstream classification and NER tasks.
Architectural Patterns and Design Principles
• Separation of Concerns: The repository is cleanly divided into two main directories (lm_pretraining and downstream_tasks) that represent distinct workflows.
• Pipeline Architecture: Emphasize the sequential pipeline for model pretraining, from raw data to formatted data to fine-tuned models.
• Reusability and Modularity: Downstream tasks act as example applications showing how the pre-trained models can be integrated into various clinical NLP tasks.
• External libraries or frameworks if needed—most notably the usage of Hugging Face’s Transformers library for model deployment (as mentioned in the README).
• For completeness, add a note for environment setup such as dependencies coming from requirements.txt and any configuration inferred from the file structure.