# healthylaife/mimic-iv-data-pipeline architecture

> Architecture diagram of the GitHub repository healthylaife/mimic-iv-data-pipeline, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/healthylaife/mimic-iv-data-pipeline
- Repository: https://github.com/healthylaife/mimic-iv-data-pipeline

## Overview

This repository is a configurable MIMIC-IV preparation and downstream-analysis pipeline. Its user-facing workflow, documented around `mainPipeline.ipynb`, selects a cohort and features, processes structured and clinical-text data into time-aligned datasets, and supports modeling and evaluation. The README also describes multimodal extensions for notes, imaging, ECG, echo, and waveforms; their embedding notebooks are represented as a distinct branch. The graph combines README-described capabilities with sampled source behavior; unsampled subsystem wiring is shown only where documented.

Main entry points: [`mainPipeline.ipynb`](https://github.com/healthylaife/mimic-iv-data-pipeline/blob/main/mainPipeline.ipynb) (Workflow notebook), [`data`](https://github.com/healthylaife/mimic-iv-data-pipeline/tree/main/data) (Processed features), [`preprocessing/hosp_module_preproc/feature_selection_hosp.py`](https://github.com/healthylaife/mimic-iv-data-pipeline/blob/main/preprocessing/hosp_module_preproc/feature_selection_hosp.py) (Hospital features), [`preprocessing/hosp_module_preproc/feature_selection_icu.py`](https://github.com/healthylaife/mimic-iv-data-pipeline/blob/main/preprocessing/hosp_module_preproc/feature_selection_icu.py) (ICU features)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_entry["Pipeline entry"]
  node_notebook["Workflow notebook<br/>[mainPipeline.ipynb]"]
end

subgraph group_cohort["Cohort preparation"]
  node_cohortgen["Cohort extraction"]
  node_cohortdata["Cohort files"]
end

subgraph group_features["Feature processing"]
  node_hospfeat["Hospital features"]
  node_icufeat["ICU features"]
  node_hosphelpers["Hospital preprocessing"]
  node_icuhelpers["ICU preprocessing"]
  node_cleaning["Cleaning and conversion"]
  node_notes["Clinical notes<br/>[notes_preproc.py]"]
  node_intervals["Time-series assembly<br/>[data_generation.py]"]
  node_icuintervals["ICU time-series"]
  node_featuredata["Processed features"]
end

subgraph group_multimodal["Multimodal extensions"]
  node_embeddings["Modality embeddings"]
end

subgraph group_analysis["Modeling and evaluation"]
  node_models["Prediction models"]
  node_evaluation["Model evaluation<br/>[evaluation.py]"]
  node_fairness["Fairness measures<br/>[fairness.py]"]
  node_outputs["Analysis outputs"]
end

node_user(("Pipeline user"))
node_mimic["MIMIC-IV sources"]

node_user -->|"configures workflow"| node_notebook
node_mimic -->|"provides input data"| node_notebook
node_notebook -->|"extracts cohort"| node_cohortgen
node_cohortgen -->|"writes cohort"| node_cohortdata
node_notebook -->|"selects features"| node_hospfeat
node_notebook -->|"selects features"| node_icufeat
node_hospfeat -->|"calls preprocessing"| node_hosphelpers
node_icufeat -->|"calls preprocessing"| node_icuhelpers
node_hospfeat -->|"cleans and groups"| node_cleaning
node_icufeat -->|"cleans and groups"| node_cleaning
node_notebook -.->|"preprocesses notes"| node_notes
node_notes -->|"produces note data"| node_featuredata
node_hospfeat -->|"writes features"| node_featuredata
node_icufeat -->|"writes features"| node_featuredata
node_intervals -->|"reads cohort"| node_cohortdata
node_intervals -->|"reads features"| node_featuredata
node_icuintervals -->|"reads cohort"| node_cohortdata
node_icuintervals -->|"reads features"| node_featuredata
node_notebook -->|"generates sequences"| node_intervals
node_notebook -->|"generates sequences"| node_icuintervals
node_mimic -.->|"supplies modalities"| node_embeddings
node_embeddings -.->|"produces embeddings"| node_featuredata
node_notebook -->|"supports modeling"| node_models
node_models -->|"evaluated by"| node_evaluation
node_evaluation -->|"reports fairness"| node_fairness
node_models -->|"produces results"| node_outputs
node_evaluation -->|"writes metrics"| node_outputs

click node_notebook "https://github.com/healthylaife/mimic-iv-data-pipeline/blob/main/mainPipeline.ipynb"
click node_cohortgen "https://github.com/healthylaife/mimic-iv-data-pipeline/blob/main/preprocessing/day_intervals_preproc/day_intervals_cohort_v3.py"
click node_cohortdata "https://github.com/healthylaife/mimic-iv-data-pipeline/tree/main/data"
click node_hospfeat "https://github.com/healthylaife/mimic-iv-data-pipeline/blob/main/preprocessing/hosp_module_preproc/feature_selection_hosp.py"
click node_icufeat "https://github.com/healthylaife/mimic-iv-data-pipeline/blob/main/preprocessing/hosp_module_preproc/feature_selection_icu.py"
click node_hosphelpers "https://github.com/healthylaife/mimic-iv-data-pipeline/blob/main/utils/hosp_preprocess_util.py"
click node_icuhelpers "https://github.com/healthylaife/mimic-iv-data-pipeline/blob/main/utils/icu_preprocess_util.py"
click node_cleaning "https://github.com/healthylaife/mimic-iv-data-pipeline/tree/main/utils"
click node_notes "https://github.com/healthylaife/mimic-iv-data-pipeline/blob/main/preprocessing/notes_preproc.py"
click node_intervals "https://github.com/healthylaife/mimic-iv-data-pipeline/blob/main/model/data_generation.py"
click node_icuintervals "https://github.com/healthylaife/mimic-iv-data-pipeline/blob/main/model/data_generation_icu.py"
click node_featuredata "https://github.com/healthylaife/mimic-iv-data-pipeline/tree/main/data"
click node_embeddings "https://github.com/healthylaife/mimic-iv-data-pipeline/tree/main/multimodal-pipeline/Multimodal-Embeddings"
click node_models "https://github.com/healthylaife/mimic-iv-data-pipeline/tree/main/model"
click node_evaluation "https://github.com/healthylaife/mimic-iv-data-pipeline/blob/main/model/evaluation.py"
click node_fairness "https://github.com/healthylaife/mimic-iv-data-pipeline/blob/main/model/fairness.py"
click node_outputs "https://github.com/healthylaife/mimic-iv-data-pipeline/tree/main/data"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_notebook,node_user toneBlue
class node_cohortgen,node_cohortdata toneAmber
class node_hospfeat,node_icufeat,node_hosphelpers,node_icuhelpers,node_cleaning,node_notes,node_intervals,node_icuintervals,node_featuredata toneMint
class node_embeddings toneRose
class node_models,node_evaluation,node_fairness,node_outputs,node_mimic toneIndigo
```

## Components

### Pipeline entry

- [Workflow notebook](https://github.com/healthylaife/mimic-iv-data-pipeline/blob/main/mainPipeline.ipynb): `mainPipeline.ipynb`

### Cohort preparation

- [Cohort extraction](https://github.com/healthylaife/mimic-iv-data-pipeline/blob/main/preprocessing/day_intervals_preproc/day_intervals_cohort_v3.py): `preprocessing/day_intervals_preproc/day_intervals_cohort_v3.py`
- [Cohort files](https://github.com/healthylaife/mimic-iv-data-pipeline/tree/main/data): `data`

### Feature processing

- [Hospital features](https://github.com/healthylaife/mimic-iv-data-pipeline/blob/main/preprocessing/hosp_module_preproc/feature_selection_hosp.py): `preprocessing/hosp_module_preproc/feature_selection_hosp.py`
- [ICU features](https://github.com/healthylaife/mimic-iv-data-pipeline/blob/main/preprocessing/hosp_module_preproc/feature_selection_icu.py): `preprocessing/hosp_module_preproc/feature_selection_icu.py`
- [Hospital preprocessing](https://github.com/healthylaife/mimic-iv-data-pipeline/blob/main/utils/hosp_preprocess_util.py): `utils/hosp_preprocess_util.py`
- [ICU preprocessing](https://github.com/healthylaife/mimic-iv-data-pipeline/blob/main/utils/icu_preprocess_util.py): `utils/icu_preprocess_util.py`
- [Cleaning and conversion](https://github.com/healthylaife/mimic-iv-data-pipeline/tree/main/utils): `utils`
- [Clinical notes](https://github.com/healthylaife/mimic-iv-data-pipeline/blob/main/preprocessing/notes_preproc.py): `preprocessing/notes_preproc.py`
- [Time-series assembly](https://github.com/healthylaife/mimic-iv-data-pipeline/blob/main/model/data_generation.py): `model/data_generation.py`
- [ICU time-series](https://github.com/healthylaife/mimic-iv-data-pipeline/blob/main/model/data_generation_icu.py): `model/data_generation_icu.py`
- [Processed features](https://github.com/healthylaife/mimic-iv-data-pipeline/tree/main/data): `data`

### Multimodal extensions

- [Modality embeddings](https://github.com/healthylaife/mimic-iv-data-pipeline/tree/main/multimodal-pipeline/Multimodal-Embeddings): `multimodal-pipeline/Multimodal-Embeddings`

### Modeling and evaluation

- [Prediction models](https://github.com/healthylaife/mimic-iv-data-pipeline/tree/main/model): `model`
- [Model evaluation](https://github.com/healthylaife/mimic-iv-data-pipeline/blob/main/model/evaluation.py): `model/evaluation.py`
- [Fairness measures](https://github.com/healthylaife/mimic-iv-data-pipeline/blob/main/model/fairness.py): `model/fairness.py`
- [Analysis outputs](https://github.com/healthylaife/mimic-iv-data-pipeline/tree/main/data): `data`

### Other components

- **Pipeline user**
- **MIMIC-IV sources**

## Connections

- Pipeline user → Workflow notebook: configures workflow
- MIMIC-IV sources → Workflow notebook: provides input data
- Workflow notebook → Cohort extraction: extracts cohort
- Cohort extraction → Cohort files: writes cohort
- Workflow notebook → Hospital features: selects features
- Workflow notebook → ICU features: selects features
- Hospital features → Hospital preprocessing: calls preprocessing
- ICU features → ICU preprocessing: calls preprocessing
- Hospital features → Cleaning and conversion: cleans and groups
- ICU features → Cleaning and conversion: cleans and groups
- Workflow notebook → Clinical notes: preprocesses notes
- Clinical notes → Processed features: produces note data
- Hospital features → Processed features: writes features
- ICU features → Processed features: writes features
- Time-series assembly → Cohort files: reads cohort
- Time-series assembly → Processed features: reads features
- ICU time-series → Cohort files: reads cohort
- ICU time-series → Processed features: reads features
- Workflow notebook → Time-series assembly: generates sequences
- Workflow notebook → ICU time-series: generates sequences
- MIMIC-IV sources → Modality embeddings: supplies modalities
- Modality embeddings → Processed features: produces embeddings
- Workflow notebook → Prediction models: supports modeling
- Prediction models → Model evaluation: evaluated by
- Model evaluation → Fairness measures: reports fairness
- Prediction models → Analysis outputs: produces results
- Model evaluation → Analysis outputs: writes metrics