How karenwhite33/diagnovision_radiology_assistant works
Project Type and Purpose
• This is a full-stack, multimodal AI application (“DiagnoVision”) for automated chest X-ray analysis and report generation.
• It combines computer vision (Vision Transformer), natural language processing (Random Forest classifier + LLM), model fusion, and exposes a REST/Gradio API for inference and deployment on Hugging Face and Google Cloud Run.
File Structure
• 1_1img_preprocessing, 1_data_preprocessing
– Image- and text-level data cleaning, splitting, and normalization notebooks and scripts.
• 2_2img_model, 2_text_model
– Jupyter notebooks for training the ViT image classifier and the RandomForest text classifier.
• 3_fusion_model
– Notebook that merges image and text model outputs and refines predictions.
• 4_docker_fastAPI_HuatuoGPT
– FastAPI application code (apiapp.py, gradio_apiapp.py), model loading (vit_inf.py, load_rf_model.py), and fusion logic. Intended to run in a Docker container.
• 5.deployment_HUGGINGFACE
– A second inference app (app.py, apiapp4.py) plus requirements for deploying to a Hugging Face Space.
• data, samples
– CSVs for model inputs and example X-ray images with associated text for demo.
README for Additional Insights
• Phases: Data Preparation → Model Training → Fusion & Optimization → Evaluation → FastAPI Deployment
• Technology Stack: google/vit-base-patch16-224; RandomForestClassifier; FreedomIntelligence/HuatuoGPT-o1-7B LLM
• Deployment targets: Docker + Google Cloud Run, Hugging Face Spaces
a. Main Components
1. Data Ingestion & Preprocessing
– Image prep scripts (resize, normalize) & text tokenization/joining.
2. Model Training Modules
– Vision Transformer training pipeline (images → ViT)
– Random Forest text classifier pipeline (reports → RF)
3. Model Fusion Module
– Combines ViT outputs + RF probabilities → unified pathology vector
4. LLM Report Generator
– HuatuoGPT-7B takes fused outputs → structured radiology report
5. Inference Layer (FastAPI / Gradio)
– REST/HTTP endpoints for image upload, classification, report generation
6. Deployment Infrastructure
– Docker container on Google Cloud Run; Hugging Face Space for demo
7. External Dependencies
– CheXpert dataset for training; cloud services (Google Cloud Run); HF Hub
b. Relationships & Data Flow
• Raw X-ray image → Image Preprocessing → ViT Model → Image Pathology Scores
• Raw Text (optional) → Text Preprocessing → RF Model → Text Pathology Scores
• Two score vectors → Fusion Module → Final Pathology Vector → LLM → Report
• Inference API receives image/text → routes through above → returns JSON + report
c. Architectural Patterns & Principles
• Modular ML Pipeline: clear separation of preprocessing, training, fusion, inference
• Microservice-style Inference: FastAPI acts as a service layer decoupled from model code
• Containerized Deployment: Docker + Cloud Run for scalability and reproducibility
• TensorFlow/PyTorch (ViT), scikit-learn (RandomForest), Hugging Face Transformers (LLM)
• FastAPI & Uvicorn for REST, Gradio for interactive UI
• Docker, Google Cloud Run, Hugging Face Spaces