How ahmedmostafaattiaaa/dual-ai-workbench-llava-vision-deepseek-code works
Project Type & Purpose
• This is a single‐process, Python‐based Gradio web application that unifies two AI functions:
– Image Description (multimodal LLaVA‐NEXT model)
– Code Generation (Qwen‐1.5B distilled model + Flake8 linting)
• No external database/service is required beyond the Hugging Face Hub.
File Structure
Top Level:
• app.py – main entry point that builds & launches the Gradio interface
• requirements.txt – Python dependencies
• assets/, notebooks/, external_files/ – ancillary materials (screenshots, reference notebooks, PDF)
src/ directory (core logic & UI components):
• config.py – central configuration (model names, device selection, default prompts)
• code_generator/
– core.py – model loading & generation logic + Flake8 lint
– interface.py – Gradio components for “Code Generator” tab
• image_descriptor/
– core.py – model loading & image inference logic
– interface.py – Gradio components for “Image Description” tab
• utils/ – any shared utility functions
Notebooks and EXTERNAL_Notebooks – reference, not part of production
Extract Architectural Insights from README
• Two tabbed Gradio interfaces sharing one web server
• Models are loaded sequentially onto GPU (4‐bit via bitsandbytes) or CPU/MPS fallback
• Hugging Face Hub serves as external model repository
• UI components in interface.py modules call respective core logic functions
Main Components & Interactions
A. Frontend (Gradio Web App)
– app.py orchestrates:
• Layout: three tabs – Image Description, Code Generator, About Me
• Binds UI controls (buttons, sliders, file upload) to backend functions
B. Backend Modules
– src/config.py: global settings (DEVICE, model IDs, default prompts)
– src/image_descriptor/core.py:
• Load llava‐hf model via Hugging Face & bitsandbytes
• Preprocess image, run inference, postprocess text output
– src/code_generator/core.py:
• Load DeepSeek Qwen model via HF & bitsandbytes
• Generate code, run Flake8 lint, return code + lint report
C. External Dependencies & Services
– Hugging Face Model Hub (model downloads & caching)
– bitsandbytes (quantization)
– PyTorch (model execution on GPU/CPU/MPS)
– Flake8 (code quality)
D. Hardware Resource
– GPU with CUDA (preferred), MPS (Mac), or CPU
– VRAM and system RAM
Architectural Patterns & Design Principles
• Modular separation: interface vs core logic vs configuration
• Single entry point (app.py) keeps launch logic centralized
• Dependency injection via config.py for easy swapping of models or device
• Sequential model loading with failure detection & UI feedback
• Tabbed UI isolates business logic per feature