# csinva/clinical-rule-development architecture

> Architecture diagram of the GitHub repository csinva/clinical-rule-development, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-01). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/csinva/clinical-rule-development
- Repository: https://github.com/csinva/clinical-rule-development

## Overview

##### Project Type and Purpose
- This is a monorepo hosting three related but independent research pipelines (“analyzing_patient_perspectives”, “cdi_survey”, “iai_vetting”), plus an “experimental” folder.
- Each subproject implements a data-driven clinical decision rules (CDI) lifecycle: raw data ingestion, cleaning/processing, analysis (statistical, machine learning, LLM), and results generation (figures, tables, reports).

##### High-Level Architecture Pattern
- All three subprojects follow essentially the same logical pipeline pattern:
- Data Sources (public datasets, PubMed metadata, eCRFs, JSON/CSV raw files)
- Data Preparation (notebooks, scripts that scrape/clean/transform)
- Processing & Modeling (classification, sentiment analysis, vetting models)
- Output Generation (figures in PDFs/CSV, pickled models/dataframes, HTML/PDF reports)
- Documentation (README, inline docs, summary notebooks)

##### Main Components and Their Relationships
a. Data Sources
- “data/raw” or “data/metadata” folders in each subproject (e.g. PubMed JSON in cdi_survey/data/metadata; CSV exports in iai_vetting/data/pecarn).
- External services: PubMed API, MDCalc scraping, exception-from-consent surveys.
b. Data Preparation
- Jupyter notebooks under “notebooks_data_prep” or top-level “notebooks” (Python code + .py helpers).
- Scripts (e.g. cdi_survey/notebooks_data_prep/*.py, data.py modules) that normalize, filter, merge raw data into cleaned CSV/JSON.
c. Processing & Modeling
- Analysis notebooks (e.g. analyzing_patient_perspectives/notebooks/*.ipynb, cdi_survey/notebooks_llm, iai_vetting/src/train.py & validate.py).
- Code modules (iai_vetting/src/*.py, sentiment/theme classification scripts).
- Machine learning libraries (scikit-learn for classification, Llama/GPT calls for theme generation).
d. Processed Artifacts
- Pickle files (…/processed/*.pkl), JSON outputs (…/data/processed/*.json), model artifacts.
e. Results & Visualization
- Figures in PDF (…/figs/*.pdf), results CSV (…/results/*.csv), HTML reports (iai_vetting/docs).
- Build scripts for documentation (docs/build_docs.sh).
f. Documentation & Reports
- Subproject READMEs, high-level readme.md, auto-generated HTML docs (iai_vetting/docs/*.html).

##### Important Architectural Principles
- Clear separation of concerns: raw vs processed data, data-prep vs analysis vs reporting.
- Reproducibility: notebooks with explicit order (00_.., 01_..), script wrappers for batch runs.
- Modularity: Python modules under src/ for reusable functions (data loading, viz, training).
- Version control: all notebooks, scripts, data folders tracked in Git (with .gitignore).

## Diagram (Mermaid)

```mermaid
flowchart TD
    subgraph "Data Ingestion"
        direction TB
        AP_raw["Analyzing Patient Perspectives\nRaw Data"]:::datastore
        CSI_data["CDI Survey\nData & Metadata"]:::datastore
        IV_pecarn["IAI Vetting\nPecarn Raw Data"]:::datastore
        IV_psrc["IAI Vetting\nPSRC Raw Data"]:::datastore
        IV_tbi["IAI Vetting\nTBI Raw Data"]:::datastore
        PubMedAPI["PubMed API"]:::external
        MDCalc["MDCalc Scraper"]:::external
    end

    subgraph "Data Preparation"
        direction TB
        AP_nb0["00_sentiment_run.ipynb"]:::notebook
        AP_py_data["data.py"]:::code
        AP_py_theme["theme_classification.py"]:::code

        CSI_nb_prep["CDI Prep Notebooks\n(*.ipynb)"]:::notebook
        CSI_py_mdcalc["mdcalc.py"]:::code
        CSI_py_paper["paper_parsing.py"]:::code
        CSI_py_pub["pubmed.py"]:::code
        CSI_py_viz["viz.py"]:::code

        IV_prep["process_basic.py"]:::code
    end

    subgraph "Processing & Modeling"
        direction TB
        AP_proc_nb["AP Analysis Notebooks"]:::notebook

        CSI_llm_nb["CDI LLM Notebooks\n(*.ipynb)"]:::notebook
        CSI_py_class["02_classification.py"]:::code
        CSI_py_extr["extraction.py"]:::code
        CSI_py_feat["feat_select.py"]:::code

        IV_train["train.py"]:::code
        IV_validate["validate.py"]:::code
        IV_viz_module["viz.py"]:::code
        IV_src_data["data*.py"]:::code
        IV_notebooks["IAI Analysis Notebooks\n(*.ipynb)"]:::notebook
        IV_nb_pecarn["pecarn_predict.ipynb"]:::notebook
    end

    subgraph "Outputs"
        direction TB
        AP_artifacts["Processed Artifacts\n(*.pkl)"]:::datastore
        AP_figs["Figures (PDF)"]:::output

        CSI_artifacts["Processed Artifacts\n(*.json/*.pkl)"]:::datastore
        CSI_datapkl["Data Pickles\n(*.pkl)"]:::datastore
        CSI_results["Results (CSV)"]:::output
        CSI_llm_results["LLM Results"]:::output

        IV_df["df_pecarn.csv"]:::datastore
        IV_reports["Reports (HTML/PDF)"]:::output
        IV_figs["Report Figures"]:::output
    end

    subgraph "Documentation"
        direction TB
        AP_doc["README.md"]:::doc
        CSI_doc["README.md"]:::doc
        IV_build["build_docs.sh"]:::code
        IV_html["HTML Docs"]:::doc
        IV_src_doc["documentation.md"]:::doc
    end

    %% Data Ingestion -> Data Preparation
    AP_raw --> AP_nb0
    AP_raw --> AP_py_data
    AP_raw --> AP_py_theme
    CSI_data --> CSI_nb_prep
    CSI_data --> CSI_py_mdcalc
    CSI_data --> CSI_py_paper
    CSI_data --> CSI_py_pub
    PubMedAPI --> CSI_py_pub
    MDCalc --> CSI_py_mdcalc
    IV_pecarn --> IV_prep
    IV_psrc --> IV_prep
    IV_tbi --> IV_prep

    %% Data Preparation -> Processing & Modeling
    AP_nb0 --> AP_proc_nb
    AP_py_data --> AP_proc_nb
    AP_py_theme --> AP_proc_nb

    CSI_nb_prep --> CSI_llm_nb
    CSI_py_mdcalc --> CSI_llm_nb
    CSI_py_paper --> CSI_llm_nb
    CSI_py_pub --> CSI_llm_nb
    CSI_py_viz --> CSI_llm_nb

    IV_prep --> IV_train
    IV_prep --> IV_validate
    IV_prep --> IV_notebooks

    %% Processing & Modeling -> Outputs
    AP_proc_nb --> AP_artifacts
    AP_proc_nb --> AP_figs

    CSI_llm_nb --> CSI_artifacts
    CSI_llm_nb --> CSI_datapkl
    CSI_llm_nb --> CSI_results
    CSI_llm_nb --> CSI_llm_results

    IV_train --> IV_df
    IV_nb_pecarn --> IV_reports
    IV_train --> IV_figs

    %% Documentation generation
    IV_build --> IV_html

    %% Click Events
    click AP_raw "https://github.com/csinva/clinical-rule-development/tree/master/analyzing_patient_perspectives/data/raw"
    click AP_nb0 "https://github.com/csinva/clinical-rule-development/blob/master/analyzing_patient_perspectives/notebooks/00_sentiment_run.ipynb"
    click AP_py_data "https://github.com/csinva/clinical-rule-development/blob/master/analyzing_patient_perspectives/notebooks/data.py"
    click AP_py_theme "https://github.com/csinva/clinical-rule-development/blob/master/analyzing_patient_perspectives/notebooks/theme_classification.py"
    click AP_artifacts "https://github.com/csinva/clinical-rule-development/blob/master/analyzing_patient_perspectives/processed/*.pkl"
    click AP_figs "https://github.com/csinva/clinical-rule-development/tree/master/analyzing_patient_perspectives/figs/"
    click AP_doc "https://github.com/csinva/clinical-rule-development/blob/master/analyzing_patient_perspectives/readme.md"

    click CSI_data "https://github.com/csinva/clinical-rule-development/tree/master/cdi_survey/data/"
    click CSI_nb_prep "https://github.com/csinva/clinical-rule-development/blob/master/cdi_survey/notebooks_data_prep/*.ipynb"
    click CSI_py_mdcalc "https://github.com/csinva/clinical-rule-development/blob/master/cdi_survey/notebooks_data_prep/mdcalc.py"
    click CSI_py_paper "https://github.com/csinva/clinical-rule-development/blob/master/cdi_survey/notebooks_data_prep/paper_parsing.py"
    click CSI_py_pub "https://github.com/csinva/clinical-rule-development/blob/master/cdi_survey/notebooks_data_prep/pubmed.py"
    click CSI_py_viz "https://github.com/csinva/clinical-rule-development/blob/master/cdi_survey/notebooks_data_prep/viz.py"
    click CSI_llm_nb "https://github.com/csinva/clinical-rule-development/blob/master/cdi_survey/notebooks_llm/*.ipynb"
    click CSI_py_class "https://github.com/csinva/clinical-rule-development/blob/master/cdi_survey/notebooks_llm/02_classification.py"
    click CSI_py_extr "https://github.com/csinva/clinical-rule-development/blob/master/cdi_survey/notebooks_llm/extraction.py"
    click CSI_py_feat "https://github.com/csinva/clinical-rule-development/blob/master/cdi_survey/notebooks_llm/feat_select.py"
    click CSI_artifacts "https://github.com/csinva/clinical-rule-development/blob/master/cdi_survey/data/processed/*.json"
    click CSI_datapkl "https://github.com/csinva/clinical-rule-development/blob/master/cdi_survey/data/*.pkl"
    click CSI_results "https://github.com/csinva/clinical-rule-development/tree/master/cdi_survey/results/"
    click CSI_llm_results "https://github.com/csinva/clinical-rule-development/tree/master/cdi_survey/results_llm/"
    click CSI_doc "https://github.com/csinva/clinical-rule-development/blob/master/cdi_survey/readme.md"

    click IV_pecarn "https://github.com/csinva/clinical-rule-development/tree/master/iai_vetting/data/pecarn/"
    click IV_psrc "https://github.com/csinva/clinical-rule-development/tree/master/iai_vetting/data/psrc/"
    click IV_tbi "https://github.com/csinva/clinical-rule-development/tree/master/iai_vetting/data/tbi/"
    click IV_prep "https://github.com/csinva/clinical-rule-development/blob/master/iai_vetting/data/psrc/process_basic.py"
    click IV_train "https://github.com/csinva/clinical-rule-development/blob/master/iai_vetting/src/train.py"
    click IV_validate "https://github.com/csinva/clinical-rule-development/blob/master/iai_vetting/src/validate.py"
    click IV_viz_module "https://github.com/csinva/clinical-rule-development/blob/master/iai_vetting/src/viz.py"
    click IV_src_data "https://github.com/csinva/clinical-rule-development/blob/master/iai_vetting/src/data*.py"
    click IV_notebooks "https://github.com/csinva/clinical-rule-development/blob/master/iai_vetting/notebooks/*.ipynb"
    click IV_nb_pecarn "https://github.com/csinva/clinical-rule-development/blob/master/iai_vetting/notebooks/pecarn_predict.py"
    click IV_df "https://github.com/csinva/clinical-rule-development/blob/master/iai_vetting/data/processed/df_pecarn.csv"
    click IV_reports "https://github.com/csinva/clinical-rule-development/tree/master/iai_vetting/reports/"
    click IV_figs "https://github.com/csinva/clinical-rule-development/tree/master/iai_vetting/reports/figs/"
    click IV_build "https://github.com/csinva/clinical-rule-development/blob/master/iai_vetting/docs/build_docs.sh"
    click IV_html "https://github.com/csinva/clinical-rule-development/blob/master/iai_vetting/docs/*.html"
    click IV_src_doc "https://github.com/csinva/clinical-rule-development/blob/master/iai_vetting/src/documentation.md"

    %% Styles
    classDef datastore fill:#bbdefb,stroke:#1e88e5
    classDef code fill:#c8e6c9,stroke:#388e3c
    classDef notebook fill:#ffe0b2,stroke:#f57c00
    classDef output fill:#e1bee7,stroke:#8e24aa
    classDef doc fill:#f5f5f5,stroke:#9e9e9e
    classDef external fill:#eeeeee,stroke:#616161
```
