# ohdsi/mimic architecture

> Architecture diagram of the GitHub repository ohdsi/mimic, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-06). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/ohdsi/mimic
- Repository: https://github.com/ohdsi/mimic

## Overview

##### Project type and purpose
- This is a data-engineering/ETL project that implements a complete pipeline to convert the MIMIC-IV clinical dataset into the OMOP Common Data Model (CDM) format.
- It pulls source data (MIMIC-IV tables and waveform CSVs), applies a multi-stage ETL (snapshot → clean → map → join → load), runs tests/metrics, then unloads the final CDM & vocabulary tables for use in OHDSI ATLAS or downstream analysis.

##### High-level components
A. Data sources
1. MIMIC-IV source tables (BigQuery or Postgres/PhysioNet)
2. Waveform CSVs on Google Cloud Storage
3. Standard OMOP vocabulary tables (loaded from Athena)
4. Custom mapping CSVs (in custom_mapping_csv, crosswalk_csv)
B. Orchestration & Config
- conf/*.etlconf & conf/workflow_*.conf drive workflows
- Python orchestrators:
– scripts/run_workflow.py (runs sets of SQL tasks)
– scripts/bq_run_script.py (executes individual SQL to BigQuery)
– scripts/wf_read.py (waveform ingestion)
C. Storage & staging datasets (BigQuery)
- ETL dataset: contains staging tables (src_*), lookup/clean/concept/mapped tables (lk_*), and CDM target tables (cdm_*)
- Vocabulary dataset: holds standard + custom vocabularies
- Final ATLAS dataset: cleansed OMOP CDM tables with technical prefixes removed
- Metrics & QA datasets: hold test results and metrics reports
D. ETL logic modules
1. ddl – DDL scripts to create empty CDM and vocabulary tables
2. staging – snapshot of source & vocabulary tables into src_* and voc_*
3. etl – core ETL:
a. clean (lk_*_clean)
b. concept mapping (lk_*_concept)
c. join & map (lk_*_mapped)
d. distribute into cdm_* tables
4. unload – unload scripts that copy cdm_* and vocabulary tables into the ATLAS dataset (removing unit_id, load_row_id, etc.)
E. Testing & reporting
– ut (unit tests on CDM tables)
– metrics_gen (generates row counts, mapping rates, custom reports)
– qa (SQL-based quality‐assurance checks)

##### Relationships and data flow
1. Source data → staging workflow
– Snapshot source and vocabulary → src_*/voc_* tables
2. Clean & map → ETL workflow
– src_* → lk_*_clean → lk_*_concept → lk_*_mapped → cdm_*
3. Vocabulary refresh → vocabulary dataset
– custom_mapping_csv & standard CSVs → load_to_bq_vocab.py → vocabulary tables
4. Waveform ingest → wf_read.py → wf_header/wf_details → used by ETL measurement & condition_occurrence
5. Testing & metrics
– After ETL, scripts/run_workflow.py with workflow_ut.conf & workflow_metrics.conf → ut_/me_ tables
6. Unload
– cdm_* & vocabulary tables → unload scripts → Atlas dataset
7. Delivery (optional)
– export_from_bq.py & load_to_bq.py → CSV export/import for external systems

##### Architectural patterns & design principles
- Layered/pipelined ETL: clearly separated stages (snapshot, clean, map, load, unload)
- Config-driven orchestration: workflows defined via conf files; Python wrappers handle execution order and error handling
- Modular SQL code: each CDM table ETL in its own SQL file (etl/etl/*.sql)
- Immutable staging: source snapshots never overwritten; intermediate tables have prefixes to track lineage
- Test & metrics integration: built directly into pipeline to enforce data quality

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data Sources
    subgraph "Data Sources"
        direction TB
        MIMIC["MIMIC-IV Source Tables"]:::source
        WAVEFORM_CSV["Waveform CSVs (GCS)"]:::source
        STD_VOCAB_CSV["Standard OMOP Vocabulary CSVs"]:::source
        CUSTOM_MAP_CSV["Custom Mapping CSVs"]:::source
    end

    %% Vocabulary Refresh Module
    subgraph "Vocabulary Refresh Module"
        direction TB
        VOC_REFRESH_PY["vocabulary_refresh.py"]:::transform
        VOC_LOAD_PY["load_to_bq_vocab.py"]:::transform
        VOC_SQL["vocabulary_refresh SQL"]:::transform
        VOC_CONF["vocabulary_refresh.conf"]:::transform
    end

    %% Waveform Ingestion
    WF_INGEST["Waveform Ingestion (wf_read.py)"]:::transform

    %% Orchestration Layer
    subgraph "Orchestration Layer"
        direction TB
        CONF["Workflow Configs (conf/*.etlconf, workflow_*.conf)"]:::orchestration
        RUN_WORKFLOW["run_workflow.py"]:::orchestration
        BQ_EXEC["bq_run_script.py"]:::orchestration
    end

    %% ETL Dataset in BigQuery
    subgraph "ETL Dataset (BigQuery)" 
        direction TB
        subgraph "Staging Schema"
            SRC_TABLES[("src_*, voc_*")]:::storage
        end
        subgraph "Lookup & Clean"
            LK_CLEAN["lk_*_clean"]:::transform
            LK_CONCEPT["lk_*_concept"]:::transform
            LK_MAPPED["lk_*_mapped"]:::transform
        end
        subgraph "CDM Target"
            CDM_TABLES[("cdm_* tables")]:::storage
        end
    end

    %% DDL Scripts
    DDL_CDM["ddl_cdm_5_3_1.sql"]:::transform
    DDL_VOC["ddl_voc_5_3_1.sql"]:::transform

    %% Unload to Atlas
    subgraph "Unload Module"
        UNLOAD_SQL["unload_to_atlas_*.sql"]:::transform
    end

    AtlasDS[("Atlas Dataset")]:::final

    %% Testing & Metrics
    subgraph "Testing & Metrics"
        UT["Unit Tests (test/ut/)"]:::testing
        METRICS["Metrics Gen (test/metrics_gen/)"]:::testing
        QA["QA Checks (test/qa/)"]:::testing
    end

    %% Delivery
    subgraph "Delivery Scripts"
        EXPORT["export_from_bq.py"]:::final
        LOAD_DELIV["load_to_bq.py"]:::final
        CDM_SCHEMAS["cdm_schemas/"]:::final
    end

    %% Data Flows
    MIMIC -->|snapshot| SRC_TABLES
    STD_VOCAB_CSV -->|load vocab| VOC_REFRESH_PY
    CUSTOM_MAP_CSV -->|load vocab| VOC_REFRESH_PY
    VOC_REFRESH_PY -->|vocab tables| SRC_TABLES
    WAVEFORM_CSV -->|ingest| WF_INGEST

    WF_INGEST --> SRC_TABLES
    CONF --> RUN_WORKFLOW
    CONF --> BQ_EXEC
    RUN_WORKFLOW -->|execute DDL| DDL_CDM
    RUN_WORKFLOW -->|execute DDL| DDL_VOC
    RUN_WORKFLOW -->|run staging| SRC_TABLES
    RUN_WORKFLOW -->|run lookup| LK_CLEAN
    BQ_EXEC --> LK_CLEAN
    LK_CLEAN --> LK_CONCEPT
    LK_CONCEPT --> LK_MAPPED
    LK_MAPPED --> CDM_TABLES
    CDM_TABLES --> UNLOAD_SQL
    UNLOAD_SQL --> AtlasDS
    CDM_TABLES --> UT
    CDM_TABLES --> METRICS
    CDM_TABLES --> QA
    AtlasDS --> EXPORT
    EXPORT --> LOAD_DELIV
    LOAD_DELIV --> CDM_SCHEMAS

    %% Click Events
    click WAVEFORM_CSV "https://github.com/ohdsi/mimic/tree/main/waveforms/"
    click STD_VOCAB_CSV "https://github.com/ohdsi/mimic/tree/main/vocabulary_refresh/omop_schemas_vocab_bq/"
    click CUSTOM_MAP_CSV "https://github.com/ohdsi/mimic/tree/main/custom_mapping_csv/"
    click CONF "https://github.com/ohdsi/mimic/tree/main/conf/"
    click RUN_WORKFLOW "https://github.com/ohdsi/mimic/blob/main/scripts/run_workflow.py"
    click BQ_EXEC "https://github.com/ohdsi/mimic/blob/main/scripts/bq_run_script.py"
    click WF_INGEST "https://github.com/ohdsi/mimic/blob/main/scripts/wf_read.py"
    click SRC_TABLES "https://github.com/ohdsi/mimic/tree/main/etl/staging/"
    click VOC_REFRESH_PY "https://github.com/ohdsi/mimic/blob/main/vocabulary_refresh/vocabulary_refresh.py"
    click VOC_LOAD_PY "https://github.com/ohdsi/mimic/blob/main/vocabulary_refresh/load_to_bq_vocab.py"
    click VOC_SQL "https://github.com/ohdsi/mimic/blob/main/vocabulary_refresh/vocabulary_refresh.sql"
    click VOC_CONF "https://github.com/ohdsi/mimic/blob/main/vocabulary_refresh/vocabulary_refresh.conf"
    click DDL_CDM "https://github.com/ohdsi/mimic/blob/main/etl/ddl/ddl_cdm_5_3_1.sql"
    click DDL_VOC "https://github.com/ohdsi/mimic/blob/main/etl/ddl/ddl_voc_5_3_1.sql"
    click LK_CLEAN "https://github.com/ohdsi/mimic/tree/main/etl/etl/"
    click UNLOAD_SQL "https://github.com/ohdsi/mimic/tree/main/etl/unload/"
    click UT "https://github.com/ohdsi/mimic/tree/main/test/ut/"
    click METRICS "https://github.com/ohdsi/mimic/tree/main/test/metrics_gen/"
    click QA "https://github.com/ohdsi/mimic/tree/main/test/qa/"
    click EXPORT "https://github.com/ohdsi/mimic/blob/main/scripts/delivery/export_from_bq.py"
    click LOAD_DELIV "https://github.com/ohdsi/mimic/blob/main/scripts/delivery/load_to_bq.py"
    click CDM_SCHEMAS "https://github.com/ohdsi/mimic/tree/main/scripts/delivery/cdm_schemas/"

    %% Styles
    classDef source fill:#D0E6FF,stroke:#0366d6,stroke-width:1px
    classDef orchestration fill:#FFDDC1,stroke:#d15704,stroke-width:1px
    classDef transform fill:#D2F8D2,stroke:#2e8b57,stroke-width:1px
    classDef storage fill:#FFF5B1,stroke:#d4a017,stroke-width:1px
    classDef testing fill:#FFF2CC,stroke:#c69c00,stroke-width:1px
    classDef final fill:#E6DAF8,stroke:#6f42c1,stroke-width:1px
```
