How csinva/clinical-rule-development works
Project Type and Purpose
• This is a monorepo hosting three related but independent research pipelines (“analyzing_patient_perspectives”, “cdi_survey”, “iai_vetting”), plus an “experimental” folder.
• Each subproject implements a data-driven clinical decision rules (CDI) lifecycle: raw data ingestion, cleaning/processing, analysis (statistical, machine learning, LLM), and results generation (figures, tables, reports).
High-Level Architecture Pattern
• All three subprojects follow essentially the same logical pipeline pattern:
• Data Sources (public datasets, PubMed metadata, eCRFs, JSON/CSV raw files)
• Data Preparation (notebooks, scripts that scrape/clean/transform)
• Processing & Modeling (classification, sentiment analysis, vetting models)
• Output Generation (figures in PDFs/CSV, pickled models/dataframes, HTML/PDF reports)
• Documentation (README, inline docs, summary notebooks)
Main Components and Their Relationships
a. Data Sources
• “data/raw” or “data/metadata” folders in each subproject (e.g. PubMed JSON in cdi_survey/data/metadata; CSV exports in iai_vetting/data/pecarn).
• External services: PubMed API, MDCalc scraping, exception-from-consent surveys.
b. Data Preparation
• Jupyter notebooks under “notebooks_data_prep” or top-level “notebooks” (Python code + .py helpers).
• Scripts (e.g. cdi_survey/notebooks_data_prep/*.py, data.py modules) that normalize, filter, merge raw data into cleaned CSV/JSON.
c. Processing & Modeling
• Analysis notebooks (e.g. analyzing_patient_perspectives/notebooks/*.ipynb, cdi_survey/notebooks_llm, iai_vetting/src/train.py & validate.py).
• Code modules (iai_vetting/src/*.py, sentiment/theme classification scripts).
• Machine learning libraries (scikit-learn for classification, Llama/GPT calls for theme generation).
d. Processed Artifacts
• Pickle files (…/processed/*.pkl), JSON outputs (…/data/processed/*.json), model artifacts.
e. Results & Visualization
• Figures in PDF (…/figs/*.pdf), results CSV (…/results/*.csv), HTML reports (iai_vetting/docs).
• Build scripts for documentation (docs/build_docs.sh).
f. Documentation & Reports
• Subproject READMEs, high-level readme.md, auto-generated HTML docs (iai_vetting/docs/*.html).
Important Architectural Principles
• Clear separation of concerns: raw vs processed data, data-prep vs analysis vs reporting.
• Reproducibility: notebooks with explicit order (00_.., 01_..), script wrappers for batch runs.
• Modularity: Python modules under src/ for reusable functions (data loading, viz, training).
• Version control: all notebooks, scripts, data folders tracked in Git (with .gitignore).