How bhanuprakashvangala/visionai works
Project Type and Purpose
• VisionAI is an AI-powered assistive system (a monolithic Python application) that processes image or voice inputs to provide real-time hazard alerts and natural speech responses for visually impaired users.
• It integrates computer vision, NLP, speech recognition, text-to-speech, and external APIs (Google Maps, Google News) into a single pipeline.
File Structure
• Top-level:
• README.md – overall project description
• src/ – application source code
• src/:
• README.md – module-level notes
• VisionAI.py – main entry point, orchestrates end-to-end pipeline
• visualizations.py – functions for audio analysis (SNR, spectrograms)
• No separate frontend/backend folders: all logic lives in Python modules under src/.
README Insights
• Key components described: scene understanding (BLIP-2, LLaMA-2), real-time hazard detection (Maps & News APIs), speech I/O (speech recognition & TTS), audio clarity evaluation.
• Technologies: PyTorch, OpenAI Transformers, OpenCV, skimage, Matplotlib/NumPy/SciPy, Google Speech API.
Main Components
a. User Interfaces
• Image/Video Input Module
• Voice Command Input (Speech Recognition)
b. Processing & AI Core
• Scene Understanding: BLIP-2 → LLaMA-2
• Hazard Detection: Google Maps API + Google News API
• Question Answering / Contextual Reasoning (LLaMA-2)
c. Output Modules
• Text-to-Speech (TTS) Engine
• Audio Clarity Analyzer (visualizations.py: SNR, spectrogram)
d. Data & Storage (optional)
• Temporary caches for API results or model outputs
e. External Services
• Google Maps Service
• Google News Service
• Google Speech Recognition Service
Relationships and Data Flow
1. User → [Image Input] → Scene Understanding → Raw Description
2. Raw Description → LLaMA-2 → Enhanced Description
3. User Location → Google Maps/News → Hazard Data
4. Enhanced Description + Hazard Data → LLaMA-2 QA → Text Response
5. Text Response → TTS → Audio Output
6. Audio Output → Audio Analyzer → SNR & Spectrogram Metrics → Logging/Display
• Asynchronous calls to external APIs and model inference steps.
Architectural Patterns & Design Principles
• Pipeline pattern: a linear sequence of processing stages
• Modular design: separate modules for vision, language, speech, API integration, and analysis
• Single responsibility: each component (module) has one clear function