# stomper10/chexpert architecture

> Architecture diagram of the GitHub repository stomper10/chexpert, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-08-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/stomper10/chexpert
- Repository: https://github.com/stomper10/chexpert

## Overview

##### Project Type and Purpose
- This is a machine‐learning research/experimentation pipeline for reproducing and extending the CheXpert DenseNet121 chest X-ray classification paper, implemented in PyTorch.
- It provides end-to-end support for data ingestion, preprocessing, model training, evaluation, deep ensembles, and visualization (Grad-CAM, ROC curves), all driven by a JSON configuration.

##### File Structure
- Data directories
- CheXpert-v1.0-small/ and CheXpert-v1.0/
(raw image datasets, frontal and lateral views)
- Configuration
- configuration.json
(hyperparameter and path settings)
- Core scripts
- run_preprocessing.py
- run_chexpert.py
- run_ensembles.py
- materials.py (data loader utilities)
- Output
- results/ (models, metrics, ROC plots, test probabilities)
- Notebooks
- CheXpert_DenseNet121_FL.ipynb (federated learning)
- Grad-CAM.ipynb (visual explanations)
- Ensembles folder
- ensembles/README.md (deep-ensemble orchestration)

##### README for Insights
- Architecture is pipeline-driven, config-based, with separation between:
1) data preprocessing,
2) model training + validation,
3) ensemble aggregation,
4) post-hoc analysis (Grad-CAM, ROC).
- Technologies: Python 3.6+, PyTorch 1.7.1, Jupyter, shell scripts for orchestration.

##### Main Components and Their Relationships
a. Data Layer
- Raw data directories (CheXpert-v1.0-small/original)
- Configuration file pointing at these paths
b. Preprocessing Module
- run_preprocessing.py + materials.py
- Reads raw CSVs & images → splits into train/val/test, applies policy (U-Zeros/Ones/Diff)
c. Training Module
- run_chexpert.py
- Loads preprocessed data → PyTorch DataLoader → DenseNet121 model (pre-trained optional)
- Uses hyperparameters from configuration.json
d. Evaluation & Metrics
- Computes losses, accuracies, AUC/ROC
- Saves model checkpoints (.pth.tar), test probabilities, ROC_*.png
e. Ensemble Aggregator
- run_ensembles.py + ensembles/ directory
- Consumes multiple experiment output directories → aggregates predictions → ensemble ROC
f. Visualization & Post-hoc Analysis
- Grad-CAM.ipynb (heatmaps), CheXpert_DenseNet121_FL.ipynb (federated learning demo)
g. Results Storage
- results/ directory hierarchy for each experiment and ensemble

##### Architectural Patterns & Design Principles
- Modular pipeline (each step is a separate script/notebook)
- Config-driven (single source of truth for hyperparameters and paths)
- Separation of concerns: data, training, ensemble, visualization
- Experiment reproducibility: random seeds, JSON config, output directory isolation

## Diagram (Mermaid)

```mermaid
flowchart TB
    subgraph "Data Layer"
        RawSmall["CheXpert-v1.0-small/"]:::dataStore
        RawFull["CheXpert-v1.0/"]:::dataStore
    end

    Config["configuration.json"]:::config

    Preprocessing["Preprocessing Module"]:::processing
    Preprocessed["Preprocessed Datasets"]:::dataStore

    Training["Training (DenseNet121)"]:::processing
    ModelArtifacts["Model Checkpoints & Metrics"]:::artifact

    Ensemble["Ensemble Aggregator"]:::processing
    EnsembleResults["Ensemble Results"]:::artifact

    subgraph "Visualization & Post-hoc Analysis"
        GradCam["Grad-CAM Notebook"]:::notebook
        FL["Federated Learning Notebook"]:::notebook
    end

    Compute["GPU Cluster"]:::infra

    RawSmall -->|images + CSVs| Preprocessing
    RawFull -->|images + CSVs| Preprocessing
    Config -->|hyperparams + paths| Preprocessing

    Preprocessing -->|train/val/test sets| Preprocessed
    Preprocessed -->|DataLoader| Training
    Config -->|hyperparams + paths| Training

    Training -->|state_dict + metrics| ModelArtifacts
    ModelArtifacts -->|predictions| Ensemble
    Ensemble -->|aggregated predictions| EnsembleResults

    ModelArtifacts -->|model state| GradCam
    ModelArtifacts -->|model state| FL

    Compute --> Preprocessing
    Compute --> Training
    Compute --> Ensemble

    click RawSmall "https://github.com/stomper10/chexpert/tree/master/CheXpert-v1.0-small/"
    click RawFull "https://github.com/stomper10/chexpert/tree/master/CheXpert-v1.0/"
    click Config "https://github.com/stomper10/chexpert/blob/master/configuration.json"
    click Preprocessing "https://github.com/stomper10/chexpert/blob/master/run_preprocessing.py"
    click Preprocessing "https://github.com/stomper10/chexpert/blob/master/materials.py"
    click Training "https://github.com/stomper10/chexpert/blob/master/run_chexpert.py"
    click ModelArtifacts "https://github.com/stomper10/chexpert/tree/master/results/"
    click Ensemble "https://github.com/stomper10/chexpert/blob/master/run_ensembles.py"
    click Ensemble "https://github.com/stomper10/chexpert/blob/master/ensembles/README.md"
    click GradCam "https://github.com/stomper10/chexpert/blob/master/Grad-CAM.ipynb"
    click FL "https://github.com/stomper10/chexpert/blob/master/CheXpert_DenseNet121_FL.ipynb"

    classDef dataStore fill:#ADD8E6,stroke:#333,stroke-width:1px
    classDef processing fill:#9ACD32,stroke:#333,stroke-width:1px
    classDef artifact fill:#FFA500,stroke:#333,stroke-width:1px
    classDef notebook fill:#DDA0DD,stroke:#333,stroke-width:1px
    classDef config fill:#FFEFD5,stroke:#333,stroke-width:1px
    classDef infra fill:#E0E0E0,stroke:#333,stroke-width:1px
```
