How stomper10/chexpert works
Project Type and Purpose
• This is a machine‐learning research/experimentation pipeline for reproducing and extending the CheXpert DenseNet121 chest X-ray classification paper, implemented in PyTorch.
• It provides end-to-end support for data ingestion, preprocessing, model training, evaluation, deep ensembles, and visualization (Grad-CAM, ROC curves), all driven by a JSON configuration.
File Structure
• Data directories
• CheXpert-v1.0-small/ and CheXpert-v1.0/
(raw image datasets, frontal and lateral views)
• Configuration
• configuration.json
(hyperparameter and path settings)
• Core scripts
• run_preprocessing.py
• run_chexpert.py
• run_ensembles.py
• materials.py (data loader utilities)
• Output
• results/ (models, metrics, ROC plots, test probabilities)
• Notebooks
• CheXpert_DenseNet121_FL.ipynb (federated learning)
• Grad-CAM.ipynb (visual explanations)
• Ensembles folder
• ensembles/README.md (deep-ensemble orchestration)
README for Insights
• Architecture is pipeline-driven, config-based, with separation between:
1) data preprocessing,
2) model training + validation,
3) ensemble aggregation,
4) post-hoc analysis (Grad-CAM, ROC).
• Technologies: Python 3.6+, PyTorch 1.7.1, Jupyter, shell scripts for orchestration.
Main Components and Their Relationships
a. Data Layer
• Raw data directories (CheXpert-v1.0-small/original)
• Configuration file pointing at these paths
b. Preprocessing Module
• run_preprocessing.py + materials.py
• Reads raw CSVs & images → splits into train/val/test, applies policy (U-Zeros/Ones/Diff)
c. Training Module
• run_chexpert.py
• Loads preprocessed data → PyTorch DataLoader → DenseNet121 model (pre-trained optional)
• Uses hyperparameters from configuration.json
d. Evaluation & Metrics
• Computes losses, accuracies, AUC/ROC
• Saves model checkpoints (.pth.tar), test probabilities, ROC_*.png
e. Ensemble Aggregator
• run_ensembles.py + ensembles/ directory
• Consumes multiple experiment output directories → aggregates predictions → ensemble ROC
f. Visualization & Post-hoc Analysis
• Grad-CAM.ipynb (heatmaps), CheXpert_DenseNet121_FL.ipynb (federated learning demo)
g. Results Storage
• results/ directory hierarchy for each experiment and ensemble
Architectural Patterns & Design Principles
• Modular pipeline (each step is a separate script/notebook)
• Config-driven (single source of truth for hyperparameters and paths)
• Separation of concerns: data, training, ensemble, visualization
• Experiment reproducibility: random seeds, JSON config, output directory isolation