# karenwhite33/diagnovision_radiology_assistant architecture

> Architecture diagram of the GitHub repository karenwhite33/diagnovision_radiology_assistant, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-22). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/karenwhite33/diagnovision_radiology_assistant
- Repository: https://github.com/karenwhite33/diagnovision_radiology_assistant

## Overview

##### Project Type and Purpose
- This is a full-stack, multimodal AI application (“DiagnoVision”) for automated chest X-ray analysis and report generation.
- It combines computer vision (Vision Transformer), natural language processing (Random Forest classifier + LLM), model fusion, and exposes a REST/Gradio API for inference and deployment on Hugging Face and Google Cloud Run.

##### File Structure
- **1_1img_preprocessing**, **1_data_preprocessing**
– Image- and text-level data cleaning, splitting, and normalization notebooks and scripts.
- **2_2img_model**, **2_text_model**
– Jupyter notebooks for training the ViT image classifier and the RandomForest text classifier.
- **3_fusion_model**
– Notebook that merges image and text model outputs and refines predictions.
- **4_docker_fastAPI_HuatuoGPT**
– FastAPI application code (apiapp.py, gradio_apiapp.py), model loading (vit_inf.py, load_rf_model.py), and fusion logic. Intended to run in a Docker container.
- **5.deployment_HUGGINGFACE**
– A second inference app (app.py, apiapp4.py) plus requirements for deploying to a Hugging Face Space.
- **data**, **samples**
– CSVs for model inputs and example X-ray images with associated text for demo.

##### README for Additional Insights
- Phases: Data Preparation → Model Training → Fusion & Optimization → Evaluation → FastAPI Deployment
- Technology Stack: google/vit-base-patch16-224; RandomForestClassifier; FreedomIntelligence/HuatuoGPT-o1-7B LLM
- Deployment targets: Docker + Google Cloud Run, Hugging Face Spaces

a. Main Components
1. **Data Ingestion & Preprocessing**
– Image prep scripts (resize, normalize) & text tokenization/joining.
2. **Model Training Modules**
– Vision Transformer training pipeline (images → ViT)
– Random Forest text classifier pipeline (reports → RF)
3. **Model Fusion Module**
– Combines ViT outputs + RF probabilities → unified pathology vector
4. **LLM Report Generator**
– HuatuoGPT-7B takes fused outputs → structured radiology report
5. **Inference Layer (FastAPI / Gradio)**
– REST/HTTP endpoints for image upload, classification, report generation
6. **Deployment Infrastructure**
– Docker container on Google Cloud Run; Hugging Face Space for demo
7. **External Dependencies**
– CheXpert dataset for training; cloud services (Google Cloud Run); HF Hub

b. Relationships & Data Flow
- Raw X-ray image → Image Preprocessing → ViT Model → Image Pathology Scores
- Raw Text (optional) → Text Preprocessing → RF Model → Text Pathology Scores
- Two score vectors → Fusion Module → Final Pathology Vector → LLM → Report
- Inference API receives image/text → routes through above → returns JSON + report

c. Architectural Patterns & Principles
- **Modular ML Pipeline**: clear separation of preprocessing, training, fusion, inference
- **Microservice-style Inference**: FastAPI acts as a service layer decoupled from model code
- **Containerized Deployment**: Docker + Cloud Run for scalability and reproducibility

- TensorFlow/PyTorch (ViT), scikit-learn (RandomForest), Hugging Face Transformers (LLM)
- FastAPI & Uvicorn for REST, Gradio for interactive UI
- Docker, Google Cloud Run, Hugging Face Spaces

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% External Data
    CheXpert["CheXpert Dataset"]:::external
    CloudRun["Google Cloud Run"]:::external
    HFSpace["Hugging Face Spaces"]:::external

    %% Offline Training
    subgraph "Offline Training"
        subgraph "Data Ingestion & Preprocessing"
            IP_NB["img_prep_train.ipynb"]:::preprocessing
            CHK["check_images.py"]:::preprocessing
            EXT["extrac_imgs.py"]:::preprocessing
            SPL["split_df.py"]:::preprocessing
            VER["verificacion_img_data.py"]:::preprocessing
            TXT1["Data_processed.ipynb"]:::preprocessing
            TXT2["nlpbasico_section_impression.ipynb"]:::preprocessing
        end
        subgraph "Model Training"
            VIT["VIT-Model.ipynb"]:::model
            EFF["Final efficientnet0303.ipynb"]:::model
            DNS["densenet0303.ipynb"]:::model
            RF["RandomForest-classified.ipynb"]:::model
        end
        subgraph "Model Fusion Module"
            FUSNB["LLM Fusion.ipynb"]:::fusion
            MLINK["ViT_and_Fusion_models_link"]:::fusion
        end
    end

    %% Online Inference
    subgraph "Online Inference"
        subgraph "Inference Layer (FastAPI / Gradio)"
            APIAPP["apiapp.py"]:::infra
            FUSLOAD["fusion_carga.py"]:::infra
            LRF["load_rf_model.py"]:::infra
            VITINF["vit_inf.py"]:::infra
            GRD["gradio_apiapp.py"]:::infra
        end
        subgraph "LLM Report Generator"
            QNTZ["quantizacion_freedoom.py"]:::fusion
        end
        USER["User Input\n(Image/Text)"]:::external
    end

    %% Deployment Infrastructure
    subgraph "Deployment Infrastructure"
        DOCKER["Docker Container"]:::deployment
        HFAPP["app.py"]:::deployment
        HFAPI4["apiapp4.py"]:::deployment
        REQ["requirements.txt"]:::deployment
    end

    %% Data Flow Offline
    CheXpert --> IP_NB
    CheXpert --> CHK
    CheXpert --> EXT
    CheXpert --> SPL
    CheXpert --> VER
    CheXpert --> TXT1
    CheXpert --> TXT2

    IP_NB --> VIT
    TXT1 --> RF
    TXT2 --> RF

    VIT --> FUSNB
    EFF --> FUSNB
    DNS --> FUSNB
    RF --> FUSNB

    FUSNB --> MLINK

    %% Deployment connections
    DOCKER --> APIAPP
    DOCKER --> VITINF
    DOCKER --> LRF
    DOCKER --> FUSLOAD
    DOCKER --> GRD
    DOCKER --> QNTZ

    HFSpace --> HFAPP
    HFSpace --> HFAPI4
    HFAPP --> VITINF
    HFAPP --> LRF
    HFAPP --> FUSLOAD
    HFAPP --> QNTZ

    CloudRun --> DOCKER

    %% Data Flow Online
    USER --> APIAPP
    APIAPP --> VITINF
    APIAPP --> LRF
    VITINF --> FUSLOAD
    LRF --> FUSLOAD
    FUSLOAD --> QNTZ
    QNTZ --> APIAPP
    APIAPP --> USER

    %% Click Events
    click IP_NB "https://github.com/karenwhite33/diagnovision_radiology_assistant/blob/main/1_1img_preprocessing/img_prep_train.ipynb"
    click CHK "https://github.com/karenwhite33/diagnovision_radiology_assistant/blob/main/1_1img_preprocessing/prepdata_imgs_python/check_images.py"
    click EXT "https://github.com/karenwhite33/diagnovision_radiology_assistant/blob/main/1_1img_preprocessing/prepdata_imgs_python/extrac_imgs.py"
    click SPL "https://github.com/karenwhite33/diagnovision_radiology_assistant/blob/main/1_1img_preprocessing/prepdata_imgs_python/split_df.py"
    click VER "https://github.com/karenwhite33/diagnovision_radiology_assistant/blob/main/1_1img_preprocessing/prepdata_imgs_python/verificacion_img_data.py"
    click TXT1 "https://github.com/karenwhite33/diagnovision_radiology_assistant/blob/main/1_data_preprocessing/Data_processed.ipynb"
    click TXT2 "https://github.com/karenwhite33/diagnovision_radiology_assistant/blob/main/1_data_preprocessing/nlpbasico_section_impression.ipynb"
    click VIT "https://github.com/karenwhite33/diagnovision_radiology_assistant/blob/main/2_2img_model/VIT-Model.ipynb"
    click EFF "https://github.com/karenwhite33/diagnovision_radiology_assistant/blob/main/2_2img_model/Final efficientnet0303.ipynb"
    click DNS "https://github.com/karenwhite33/diagnovision_radiology_assistant/blob/main/2_2img_model/densenet0303.ipynb"
    click RF "https://github.com/karenwhite33/diagnovision_radiology_assistant/blob/main/2_text_model/RandomForest-classified.ipynb"
    click FUSNB "https://github.com/karenwhite33/diagnovision_radiology_assistant/blob/main/3_fusion_model/LLM Fusion.ipynb"
    click MLINK "https://github.com/karenwhite33/diagnovision_radiology_assistant/tree/main/3_1models/ViT_and_Fusion_models_link"
    click QNTZ "https://github.com/karenwhite33/diagnovision_radiology_assistant/blob/main/4_docker_fastAPI_HuatuoGPT/quantizacion_freedoom.py"
    click APIAPP "https://github.com/karenwhite33/diagnovision_radiology_assistant/blob/main/4_docker_fastAPI_HuatuoGPT/apiapp.py"
    click FUSLOAD "https://github.com/karenwhite33/diagnovision_radiology_assistant/blob/main/4_docker_fastAPI_HuatuoGPT/fusion_carga.py"
    click LRF "https://github.com/karenwhite33/diagnovision_radiology_assistant/blob/main/4_docker_fastAPI_HuatuoGPT/load_rf_model.py"
    click VITINF "https://github.com/karenwhite33/diagnovision_radiology_assistant/blob/main/4_docker_fastAPI_HuatuoGPT/vit_inf.py"
    click GRD "https://github.com/karenwhite33/diagnovision_radiology_assistant/blob/main/4_docker_fastAPI_HuatuoGPT/gradio_apiapp.py"
    click HFAPP "https://github.com/karenwhite33/diagnovision_radiology_assistant/blob/main/5.deployment_HUGGINGFACE/app.py"
    click HFAPI4 "https://github.com/karenwhite33/diagnovision_radiology_assistant/blob/main/5.deployment_HUGGINGFACE/apiapp4.py"
    click REQ "https://github.com/karenwhite33/diagnovision_radiology_assistant/blob/main/5.deployment_HUGGINGFACE/requirements.txt"

    %% Styles
    classDef preprocessing fill:#cce5ff,stroke:#333,stroke-width:1px
    classDef model fill:#b2fab4,stroke:#333,stroke-width:1px
    classDef fusion fill:#ffddb3,stroke:#333,stroke-width:1px
    classDef infra fill:#d1c4e9,stroke:#333,stroke-width:1px
    classDef deployment fill:#e0e0e0,stroke:#333,stroke-width:1px
    classDef external fill:#f5f5f5,stroke:#333,stroke-width:1px
```
