How rahulsharmavishwakarma/medi-care works
Project Type and Purpose
• The README clearly identifies the project as “Medi-Care” – a system aimed at using visual question answering in the medical domain. It integrates visual data processing (via LLaVA-Med) with advanced language generation (using LLaMA) for diagnostic support.
• This makes it essentially a full-stack application focused on deep learning inference and model serving with a strong front-end for interaction (Gradio web server), along with backend processing for model training, evaluation, and data processing.
Analyzing the File Structure
• Key directories include:
• "llava": Contains the core logic for model definition, data processing, inference, training, and evaluation. Subdirectories such as "serve" indicate multiple server components (controller, model_worker, gradio_web_server, etc.) that are part of the inference pipeline.
• "data": Contains various datasets and CSV files, indicating involvement in pre-processing and fine-tuning.
• "scripts": Provides launch scripts and orchestration for training, evaluation, weights delta computation, etc.
• "playground": Likely used for quick experiments and testing.
• The split between training (llava/train), evaluation (llava/eval), serving (llava/serve), and data (data and llava/data) strongly suggests architectural segregation into data handling, model operations, and serving/inference components.
• Presence of configuration and build scripts (like run_llava.sh, run_training_llava_med.sh etc.) also help in understanding execution flows.
Extracting Key Architectural Insights from the README
• The README lists out a clear set of commands to start different inference components – the controller, the model worker, and the Gradio web server.
• This tells you that the architecture is built around a controller which manages the flow, a model worker that loads LLaVA-Med weights itself (with support for multimodal and GPU operations), and a front-end (Gradio-based) for user interaction.
• The description of installation and contribution guidelines further confirms that the system is modular, with distinct responsibilities allocated to various components in the “llava” package.
a. Main Components:
• Controller: Acts as the central orchestrator managing requests.
• Model Worker: Loads models (like LLaVA-Med) and handles the computation/inference requests.
• Gradio Web Server/UI: Provides the front-end that interacts with users.
• Data Handling / Preprocessing: Modules located in "llava/data", and additional scripts/data directories for processing CSVs, images, and annotations.
• Training & Evaluation: Services in "llava/train" and "llava/eval" that support refining and benchmark testing the models.
• Supporting Scripts: Additional helper components and scripts (e.g., in "scripts/chunyl") that automate jobs such as deploying the demo or computing delta weights for model updates.
b. Relationships and Interactions:
• The Client/User sends requests to the Gradio web server.
• The Gradio web server communicates with the Controller to dispatch or schedule requests.
• The Controller forwards requests to the Model Worker.
• The Model Worker processes the input data (e.g., images and question text) using LLaVA-Med and LLaMA models, then returns answers.
• Data pipelines feed into both training and evaluation components, which in turn inform updates or performance adjustments, possibly feeding back into the training loop.
• Optionally, external data sources or contribution code can be indicated for extensibility and community contributions.
c. Architectural Patterns and Design Principles:
• Modular Design: Clear separation of concerns between serving, training, and data preprocessing.
• Microservice-Like Structure: The use of separate processes (controller, model worker, web server) can be diagrammed as standalone services that communicate over defined APIs or network calls.
• Scalability: The design supports multiple workers or potential scaling, as indicated by command-line options like “--num-gpus” for the model worker.
d. Technologies, Frameworks, and Libraries:
• Python is the primary language – as seen from the .py files.
• Gradio is used for the web interface.
• The system leverages Git, Git LFS and standard Python libraries for managing dependencies (via requirements.txt, pyproject.toml).
• Deep learning frameworks (likely PyTorch given the style of files and conventions) for model serving and training.
• Shell scripts for orchestration indicate Unix-based deployments.