# pravatn01/visualingo-images-to-words-and-voice-with-ai architecture

> Architecture diagram of the GitHub repository pravatn01/visualingo-images-to-words-and-voice-with-ai, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/pravatn01/visualingo-images-to-words-and-voice-with-ai
- Repository: https://github.com/pravatn01/visualingo-images-to-words-and-voice-with-ai

## Overview

##### Project Type and Purpose
- Full-stack AI web application combining:
– Frontend: Gradio UI for image upload, caption display, audio playback
– Backend: Python service (webapp.py) orchestrating inference and TTS
- Two parallel ML inference pipelines:
– Baseline CNN+LSTM (TensorFlow)
– Pretrained BLIP transformer (PyTorch)

##### File Structure
- .gradio/…
– Gradio config & SSL certificate for secure hosting
- Images/…
– Sample images (optional assets)
- Models/…
– model.keras, features.pkl, tokenizer.pkl (TensorFlow pipeline artifacts)
– tokenizer.pkl (text preprocessing)
- cnn-lstm-training.ipynb
– Notebook for training baseline model on Flickr8k
- environment.yml
– Conda environment spec (TensorFlow, PyTorch, Gradio, gTTS, etc.)
- webapp.py
– Entry point: sets up Gradio interface, loads models, handles inference & TTS

##### Extract Key README Insights
- Two model types, side-by-side comparison
- Gradio UI for seamless upload→inference→playback loop
- gTTS for client-side text-to-speech conversion

##### Main Components
a. Frontend Layer
– Gradio Web UI (webapp.py)
– .gradio configuration (SSL, port)
b. Backend / Orchestration
– Python service (webapp.py)
– Loads both model pipelines & gTTS library
c. ML Inference Pipelines
1. TensorFlow: DenseNet201 feature extractor → LSTM decoder
2. PyTorch: Pretrained BLIP transformer
– Model artifacts read from Models/ directory
d. Text-to-Speech Service
– gTTS library calls Google TTS API
e. Storage & Assets
– Local filesystem volume: Models/, Images/
f. Optional Training Pipeline (cnnlstm-training notebook)

##### Relationships & Data Flow
1. User uploads image via Gradio UI →
2. webapp.py receives image →
3. Forks image to both ML pipelines in parallel →
4. Each pipeline loads model artifacts →
5. Generates caption text → returns to webapp
6. webapp.py sends caption text to gTTS → receives audio stream →
7. UI displays captions and plays audio

##### Architectural Patterns & Principles
- Monolithic Python backend with modular inference components
- Parallel processing for model comparison
- Environment-driven configuration (environment.yml, .gradio)
- File-system storage for model artifacts

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Frontend Layer
    subgraph "Frontend Layer"
        UI["Gradio Web UI"]:::frontend
        CONFIG[".gradio (SSL & Config)"]:::frontend
    end

    %% Backend Orchestrator
    subgraph "Backend Orchestrator"
        BE["webapp.py (Orchestrator & Gradio setup)"]:::backend
    end

    %% ML Inference Pipelines
    subgraph "ML Pipelines"
        CNN["CNN+LSTM TF Pipeline"]:::ml
        BLIP["BLIP Transformer Pipeline"]:::ml
    end

    %% Storage & Assets
    subgraph "Storage & Assets"
        MODELS[(Models/)]:::storage
        IMGS[(Images/)]:::storage
    end

    %% External Service
    TTS["gTTS Service (Google TTS API)"]:::external

    %% Connections
    UI -->|"Image upload"| BE
    CONFIG --> BE
    BE -->|"fork image"| CNN
    BE -->|"fork image"| BLIP
    MODELS --> CNN
    MODELS --> BLIP
    IMGS --> BE
    CNN -->|"Caption JSON"| BE
    BLIP -->|"Caption JSON"| BE
    BE -->|"Call gTTS API"| TTS
    TTS -->|"Audio stream"| BE
    BE -->|"Display caption & play audio"| UI

    %% Optional Training Pipeline
    NOTE["cnn-lstm-training.ipynb"]:::notebook
    NOTE --> CNN

    %% Environment Configuration
    ENV["environment.yml"]:::config

    classDef frontend fill:#D0E7FF,stroke:#0366d6,stroke-width:2px;
    classDef backend fill:#D3D3D3,stroke:#555,stroke-width:2px;
    classDef ml fill:#D4F7DC,stroke:#2E8B57,stroke-width:2px,stroke-dasharray: 5 5;
    classDef storage fill:#FFF3CD,stroke:#FFB800,stroke-width:2px;
    classDef external fill:#E6E6FA,stroke:#800080,stroke-width:2px;
    classDef notebook fill:#F0F0F0,stroke:#999,stroke-width:2px,stroke-dasharray: 2 2;
    classDef config fill:#F0F0F0,stroke:#999,stroke-width:2px,stroke-dasharray: 2 2;

    %% Click Events
    click UI "https://github.com/pravatn01/visualingo-images-to-words-and-voice-with-ai/blob/main/webapp.py"
    click CONFIG "https://github.com/pravatn01/visualingo-images-to-words-and-voice-with-ai/tree/main/.gradio/"
    click BE "https://github.com/pravatn01/visualingo-images-to-words-and-voice-with-ai/blob/main/webapp.py"
    click CNN "https://github.com/pravatn01/visualingo-images-to-words-and-voice-with-ai/blob/main/Models/model.keras"
    click MODELS "https://github.com/pravatn01/visualingo-images-to-words-and-voice-with-ai/tree/main/Models/"
    click IMGS "https://github.com/pravatn01/visualingo-images-to-words-and-voice-with-ai/tree/main/Images/"
    click NOTE "https://github.com/pravatn01/visualingo-images-to-words-and-voice-with-ai/blob/main/cnn-lstm-training.ipynb"
    click ENV "https://github.com/pravatn01/visualingo-images-to-words-and-voice-with-ai/blob/main/environment.yml"
    click MODELS "https://github.com/pravatn01/visualingo-images-to-words-and-voice-with-ai/blob/main/Models/features.pkl"
    click MODELS "https://github.com/pravatn01/visualingo-images-to-words-and-voice-with-ai/blob/main/Models/tokenizer.pkl"
```
