How bhimrazy/diabetic-retinopathy-detection works
Project Type and Purpose
• This is a machine‐learning pipeline with an accompanying demo web application. It ingests large ophthalmology image datasets, preprocesses them, trains diabetic-retinopathy detection models using PyTorch Lightning (with multiprocessing support), logs training via TensorBoard, and exposes inference via a Gradio web interface.
File‐Structure Analysis
• Top‐level scripts and entry points:
• app.py – launches the Gradio app for inference
• train.py – orchestrates model training
• notebook.ipynb – interactive data/transforms exploration
• Directories:
• scripts/ – data‐download and preprocessing scripts (download, merge, extract, crop_and_resize, split)
• src/ – core Python package: data_module, dataset, model definitions, concurrent_task_executor, utils, factory for model variants
• conf/ – configuration (config.yaml)
• data/ – raw and sample data
• artifacts/ – saved model checkpoints
• logs/ – TensorBoard event logs and hyperparameters
• Requirements, CI ignore, license, and readme round out the repo.
Key README Insights
• Data ingestion from Kaggle, two download modes (complete zip or parts).
• Preprocessing: cropping, resizing images in scripts.
• Training: PyTorch Lightning with configurable backbones, hyperparameters via config.yaml.
• Tracking: TensorBoard logs.
• Deployment: Gradio app (app.py) for inference using trained checkpoint.
Main Components
a. External Data Source
– Kaggle competition storage (images.zip or split parts)
b. Data-Ingestion Scripts
– scripts/download-dr-dataset.sh
– scripts/merge_and_extract.sh
c. Preprocessing Pipeline
– scripts/crop_and_resize.py (multiprocessing via concurrent_task_executor)
– scripts/split_dataset.py → outputs CSV manifest
d. Configuration
– conf/config.yaml (defines model, training params, paths)
e. Model Training Service
– train.py + src/data_module.py + src/model.py + PyTorch Lightning Trainer
– Logs metrics to logs/ (TensorBoard)
– Uses src/models/factory.py for selecting backbone
f. Artifact Store
– artifacts/dr-model.ckpt
g. Inference Web App
– app.py (Gradio interface)
– Loads checkpoint from artifacts/
– Uses src/utils.py and src/model.py to preprocess input and run inference
Architectural Patterns & Design Principles
• Separation of Concerns:
• Data ingestion & preprocessing as standalone scripts
• Training isolated in train.py using Lightning modules
• Inference in app.py
• Config-Driven:
• conf/config.yaml parameterizes paths, model hyperparams
• Multiprocessing Support:
• concurrent_task_executor for CPU‐bound tasks (crop & resize)
• Extensibility:
• src/models/factory.py allows swapping backbone architectures
• Observability:
• TensorBoard logging for metrics & hparams
Technologies & Frameworks
• Python 3.x, PyTorch Lightning, Gradio, TensorBoard
• YAML for config, Shell scripts for orchestration
• Directory‐based artifact and log management
• Break system into: Data Ingestion, Preprocessing, Training, Logging, Artifact Storage, Inference App
• Clearly call out key technologies (PyTorch Lightning, Gradio, TensorBoard) on each component
• Keep it modular—future components (new backbones, alternate UI) can slot in without redrawing core flows