# lrsoenksen/haim architecture

> Architecture diagram of the GitHub repository lrsoenksen/haim, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-30). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/lrsoenksen/haim
- Repository: https://github.com/lrsoenksen/haim

## Overview

##### Main Components
- Data Ingestion & Preprocessing:
- Initial stage where raw clinical data (from sources like MIMIC-IV and MIMIC-CXR) are gathered.
- Scripts and notebooks (files beginning with "1_") for creating pickle files and generating embeddings.
- Usage of external data sources (e.g., Physionet) and how data is prepared for later steps.
- Modeling/Training:
- Module that handles modeling (scripts labeled "2_"). There are separate scripts for mortality prediction, length-of-stay modeling, and pathology diagnosis.
- That this stage applies machine learning models on preprocessed data.
- That different clinical tasks are modeled with specific scripts and that multimodal data (tabular, time-series, text, images) is utilized.
- Evaluation & Result Generation:
- Component where analysis and visualization occur. This is represented by the "3_" series of scripts and notebooks, which produce result tables, ROC/PR curves, and calculate Shapley values.
- Diagrammatic representation of reporting (plots, metrics like AUROC, AUPRC, F1 score).
- Utility and API Components:
- Supplementary modules like MIMIC_IV_HAIM_API.py and prediction_util.py, which support data access or shared functionality.
- Environment Setup:
- A note on configuration provided by the requirements file (0-requirements.txt) and the environment specifications (env folder).

##### Relationships and Interactions
- Finally, illustrate that model outputs are passed on to the evaluation stage for performance measurement and visualization.
- Optionally, include conditional paths or dependency links indicating optional steps (e.g., handling missing "notes" data if not available).

##### Architectural Patterns and Design Principles
- Modular design of the system. Each directory or file group (0, 1, 2, 3) corresponds to a step in the pipeline.
- Separation of concerns: preprocessing, modeling, and result evaluation are clearly divided.

##### Relevant Technologies and Frameworks
- That the project is built in Python (version 3.6.9, as per README). Although not explicitly described, mention common ML libraries or data processing frameworks implicitly likely used.
- Jupyter notebooks are employed for interactive tasks; ensure to differentiate between script-based modules and notebook-based demonstration/reporting.
- The use of YAML in the env directory suggests a reliance on environment configuration and possibly Docker or Conda environments for reproducibility.

- Be very detailed: even if the implementation is script-based, show sub-components like “Data Cleaning”, “Embedding Generation”, “Model Training”, “Visualization”, and “Shapley Value Calculation” as separate modules if needed.
- Requirements/environment configuration with the actual modules, indicating that the initial setup (requirements file and env.yaml) supports all other components.

## Diagram (Mermaid)

```mermaid
graph TD
    %% Environment Setup
    subgraph "Environment Setup"
        ENV1("0-requirements.txt"):::env
        ENV2("env/env.yaml"):::env
    end

    %% External Data Source
    DS(("Physionet Data")):::data

    %% Data Ingestion & Preprocessing
    subgraph "Data Ingestion & Preprocessing"
        P1("1_1-Create Pickle Files.py"):::preprocess
        P2("1_2-Generate Embeddings.py"):::preprocess
        P3("1_Generate HAIM-MIMIC-MM from downloaded MIMIC-IV and MIMIC-CXR.ipynb"):::notebook
    end

    %% Modeling/Training
    subgraph "Modeling/Training"
        M1("2_1-Length of Stay Modeling.py"):::modeling
        M2("2_2-Mortality Prediction Modeling.py"):::modeling
        M3("2_3-Pathology Diagnosis Modeling.py"):::modeling
        M4("2_Generate Embeddings from Pickle Files.ipynb"):::notebook
    end

    %% Evaluation & Result Generation
    subgraph "Evaluation & Result Generation"
        E1("3_1-Generate Result Table for Length of Stay or Mortality Prediction.py"):::evaluation
        E2("3_2-Generate PR and ROC Result Curves for Mortality or Length of Stay.py"):::evaluation
        E3("3_3-Shapley Values Calculation.py"):::evaluation
        E4("3_Use Embedding for Prediction.ipynb"):::notebook
    end

    %% Utility and API Components
    subgraph "Utility and API Components"
        U1("MIMIC_IV_HAIM_API.py"):::utility
        U2("prediction_util.py"):::utility
    end

    %% Data Flow Connections
    DS -->|"sources"| P1
    ENV1 -->|"env_setup"| P1
    ENV2 -->|"env_setup"| P1

    %% Utility supporting Preprocessing
    U1 ---|"supports"| P1
    U2 ---|"supports"| P1

    %% Preprocessing to Modeling
    P3 -->|"preprocessed_data"| M1
    P3 -->|"preprocessed_data"| M2
    P3 -->|"preprocessed_data"| M3
    P3 -->|"preprocessed_data"| M4

    %% Modeling to Evaluation
    M1 -->|"model_output"| E1
    M2 -->|"model_output"| E2
    M3 -->|"model_output"| E3
    M4 -->|"model_output"| E4

    %% Classes for styling
    classDef env fill:#f0f0f0,stroke:#333,stroke-width:2px;
    classDef data fill:#ffccbc,stroke:#d84315,stroke-width:2px;
    classDef preprocess fill:#bbdefb,stroke:#1976d2,stroke-width:2px;
    classDef modeling fill:#c8e6c9,stroke:#388e3c,stroke-width:2px;
    classDef evaluation fill:#ffe082,stroke:#f57f17,stroke-width:2px;
    classDef utility fill:#d1c4e9,stroke:#5e35b1,stroke-width:2px;
    classDef notebook fill:#fff9c4,stroke:#f9a825,stroke-width:2px;

    %% Click Events for Data Ingestion & Preprocessing
    click P1 "https://github.com/lrsoenksen/haim/blob/main/1_1-Create Pickle Files.py"
    click P2 "https://github.com/lrsoenksen/haim/blob/main/1_2-Generate Embeddings.py"
    click P3 "https://github.com/lrsoenksen/haim/blob/main/1_Generate HAIM-MIMIC-MM from downloaded MIMIC-IV and MIMIC-CXR.ipynb"

    %% Click Events for Modeling/Training
    click M1 "https://github.com/lrsoenksen/haim/blob/main/2_1-Length of Stay Modeling.py"
    click M2 "https://github.com/lrsoenksen/haim/blob/main/2_2-Mortality Prediction Modeling.py"
    click M3 "https://github.com/lrsoenksen/haim/blob/main/2_3-Pathology Diagnosis Modeling.py"
    click M4 "https://github.com/lrsoenksen/haim/blob/main/2_Generate Embeddings from Pickle Files.ipynb"

    %% Click Events for Evaluation & Result Generation
    click E1 "https://github.com/lrsoenksen/haim/blob/main/3_1-Generate Result Table for Length of Stay or Mortality Prediction.py"
    click E2 "https://github.com/lrsoenksen/haim/blob/main/3_2-Generate PR and ROC Result Curves for Mortality or Length of Stay.py"
    click E3 "https://github.com/lrsoenksen/haim/blob/main/3_3-Shapley Values Calculation.py"
    click E4 "https://github.com/lrsoenksen/haim/blob/main/3_Use Embedding for Prediction.ipynb"

    %% Click Events for Utility and Environment Setup
    click U1 "https://github.com/lrsoenksen/haim/blob/main/MIMIC_IV_HAIM_API.py"
    click U2 "https://github.com/lrsoenksen/haim/blob/main/prediction_util.py"
    click ENV1 "https://github.com/lrsoenksen/haim/blob/main/0-requirements.txt"
    click ENV2 "https://github.com/lrsoenksen/haim/blob/main/env/env.yaml"
```
