# bhimrazy/diabetic-retinopathy-detection architecture

> Architecture diagram of the GitHub repository bhimrazy/diabetic-retinopathy-detection, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-09-05). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/bhimrazy/diabetic-retinopathy-detection
- Repository: https://github.com/bhimrazy/diabetic-retinopathy-detection

## Overview

##### Project Type and Purpose
- This is a machine‐learning pipeline with an accompanying demo web application. It ingests large ophthalmology image datasets, preprocesses them, trains diabetic-retinopathy detection models using PyTorch Lightning (with multiprocessing support), logs training via TensorBoard, and exposes inference via a Gradio web interface.

##### File‐Structure Analysis
- Top‐level scripts and entry points:
- app.py – launches the Gradio app for inference
- train.py – orchestrates model training
- notebook.ipynb – interactive data/transforms exploration
- Directories:
- scripts/ – data‐download and preprocessing scripts (download, merge, extract, crop_and_resize, split)
- src/ – core Python package: data_module, dataset, model definitions, concurrent_task_executor, utils, factory for model variants
- conf/ – configuration (config.yaml)
- data/ – raw and sample data
- artifacts/ – saved model checkpoints
- logs/ – TensorBoard event logs and hyperparameters
- Requirements, CI ignore, license, and readme round out the repo.

##### Key README Insights
- Data ingestion from Kaggle, two download modes (complete zip or parts).
- Preprocessing: cropping, resizing images in scripts.
- Training: PyTorch Lightning with configurable backbones, hyperparameters via config.yaml.
- Tracking: TensorBoard logs.
- Deployment: Gradio app (app.py) for inference using trained checkpoint.

##### Main Components
a. External Data Source
– Kaggle competition storage (images.zip or split parts)
b. Data-Ingestion Scripts
– scripts/download-dr-dataset.sh
– scripts/merge_and_extract.sh
c. Preprocessing Pipeline
– scripts/crop_and_resize.py (multiprocessing via concurrent_task_executor)
– scripts/split_dataset.py → outputs CSV manifest
d. Configuration
– conf/config.yaml (defines model, training params, paths)
e. Model Training Service
– train.py + src/data_module.py + src/model.py + PyTorch Lightning Trainer
– Logs metrics to logs/ (TensorBoard)
– Uses src/models/factory.py for selecting backbone
f. Artifact Store
– artifacts/dr-model.ckpt
g. Inference Web App
– app.py (Gradio interface)
– Loads checkpoint from artifacts/
– Uses src/utils.py and src/model.py to preprocess input and run inference

##### Architectural Patterns & Design Principles
- Separation of Concerns:
- Data ingestion & preprocessing as standalone scripts
- Training isolated in train.py using Lightning modules
- Inference in app.py
- Config-Driven:
- conf/config.yaml parameterizes paths, model hyperparams
- Multiprocessing Support:
- concurrent_task_executor for CPU‐bound tasks (crop & resize)
- Extensibility:
- src/models/factory.py allows swapping backbone architectures
- Observability:
- TensorBoard logging for metrics & hparams

##### Technologies & Frameworks
- Python 3.x, PyTorch Lightning, Gradio, TensorBoard
- YAML for config, Shell scripts for orchestration
- Directory‐based artifact and log management

- Break system into: Data Ingestion, Preprocessing, Training, Logging, Artifact Storage, Inference App
- Clearly call out key technologies (PyTorch Lightning, Gradio, TensorBoard) on each component
- Keep it modular—future components (new backbones, alternate UI) can slot in without redrawing core flows

## Diagram (Mermaid)

```mermaid
graph TD
    %% External Data Source
    Kaggle["Kaggle Competition Storage"]:::external

    %% Data Stores
    RawData[(data/raw)]:::data
    ResizedData[(data/resized)]:::data
    CSVManifest[(CSV Manifest)]:::data
    Logs[(logs/)]:::logging
    Artifacts[(artifacts/dr-model.ckpt)]:::data

    %% Configuration
    Config["conf/config.yaml"]:::process

    %% Data Ingestion
    Download["download-dr-dataset.sh"]:::process
    MergeExtract["merge_and_extract.sh"]:::process

    %% Preprocessing
    CropResize["crop_and_resize.py"]:::process
    Split["split_dataset.py"]:::process
    Executor["concurrent_task_executor.py"]:::process

    %% Model Training
    Training["train.py (Lightning Trainer)"]:::process
    DataModule["src/data_module.py"]:::process
    ModelDef["src/model.py"]:::process
    Factory["src/models/factory.py"]:::process

    %% Inference Web App
    App["app.py (Gradio App)"]:::app
    Utils["src/utils.py"]:::app
    InferenceModel["src/model.py"]:::app

    %% End User
    EndUser["End-User UI"]:::external

    %% Flow
    Kaggle -->|"images.zip or parts"| Download
    Kaggle -->|"images.zip or parts"| MergeExtract
    Download -->|".zip"| MergeExtract
    MergeExtract -->|"raw images"| RawData

    RawData -->|"raw images"| CropResize
    CropResize -->|"resized images"| ResizedData
    CropResize -->|"tasks"| Executor
    ResizedData -->|".jpg"| Split
    Split -->|"train.csv"| CSVManifest

    CSVManifest -->|"+ images"| Training
    ResizedData -->|"images"| Training
    Config -->|"params"| Training
    DataModule -->|"data loader"| Training
    ModelDef -->|"model code"| Training
    Factory -->|"backbone selection"| Training

    Training -->|"TensorBoard events"| Logs
    Training -->|"checkpoint (.ckpt)"| Artifacts

    Artifacts -->|"model checkpoint"| App
    Utils -->|"helpers"| App
    InferenceModel -->|"model code"| App
    Config -->|"params"| App

    App -->|"predictions"| EndUser

    %% Click Events
    click Download "https://github.com/bhimrazy/diabetic-retinopathy-detection/blob/main/scripts/download-dr-dataset.sh"
    click MergeExtract "https://github.com/bhimrazy/diabetic-retinopathy-detection/blob/main/scripts/merge_and_extract.sh"
    click CropResize "https://github.com/bhimrazy/diabetic-retinopathy-detection/blob/main/scripts/crop_and_resize.py"
    click Split "https://github.com/bhimrazy/diabetic-retinopathy-detection/blob/main/scripts/split_dataset.py"
    click Executor "https://github.com/bhimrazy/diabetic-retinopathy-detection/blob/main/src/concurrent_task_executor.py"
    click Config "https://github.com/bhimrazy/diabetic-retinopathy-detection/blob/main/conf/config.yaml"
    click Training "https://github.com/bhimrazy/diabetic-retinopathy-detection/blob/main/train.py"
    click DataModule "https://github.com/bhimrazy/diabetic-retinopathy-detection/blob/main/src/data_module.py"
    click ModelDef "https://github.com/bhimrazy/diabetic-retinopathy-detection/blob/main/src/model.py"
    click Factory "https://github.com/bhimrazy/diabetic-retinopathy-detection/blob/main/src/models/factory.py"
    click Artifacts "https://github.com/bhimrazy/diabetic-retinopathy-detection/blob/main/artifacts/dr-model.ckpt"
    click Logs "https://github.com/bhimrazy/diabetic-retinopathy-detection/tree/main/logs/"
    click App "https://github.com/bhimrazy/diabetic-retinopathy-detection/blob/main/app.py"
    click Utils "https://github.com/bhimrazy/diabetic-retinopathy-detection/blob/main/src/utils.py"
    click InferenceModel "https://github.com/bhimrazy/diabetic-retinopathy-detection/blob/main/src/model.py"

    %% Styles
    classDef data fill:#D0E8FF,stroke:#0288D1,color:#01579B;
    classDef process fill:#E0F7FA,stroke:#00695C,color:#004D40;
    classDef app fill:#FFF3E0,stroke:#FB8C00,color:#E65100;
    classDef external fill:#ECEFF1,stroke:#78909C,color:#546E7A;
    classDef logging fill:#F3E5F5,stroke:#8E24AA,color:#6A1B9A;
```
