# yash8745/image-caption architecture

> Architecture diagram of the GitHub repository yash8745/image-caption, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-01-20). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/yash8745/image-caption
- Repository: https://github.com/yash8745/image-caption

## Overview

##### Main Components

A. Data Layer:
- Flickr8k Dataset (Image Storage)
- Text Annotations and Metadata
- Dataset Split (Train/Test/Dev)

B. Model Architecture:
- ResNet (Image Feature Extraction)
- LSTM Network (Caption Generation)
- Embedding Layer

C. Application Layer:
- Web Interface (app.py)
- Inference Pipeline

**┌─────────────────┐    ┌─────────────────┐**
- Flickr8k      │    │  Text           │
- Image Dataset  │    │  Annotations    │
- ┘    └─────────────────┘

**┌─────────────────────────────────┐**
- Data Preprocessing Pipeline    │
- Image Processing            │
- Text Tokenization           │
- Train/Test/Dev Split        │
- ┘

**┌─────────────┐    ┌─────────────┐**
- ResNet    │ -> │   LSTM      │
- (CNN)     │    │   Network   │
- ┘    └─────────────┘

**┌─────────────────────┐**
- Flask App        │
- (app.py)        │
- ┘

##### Important Elements to Highlight

- How images flow through ResNet
- Text processing through the LSTM
- Connection between ResNet features and LSTM input

B. Processing Steps:
- Image preprocessing
- Text tokenization
- Feature extraction
- Caption generation

C. Integration Points:
- Dataset loading
- Model inference
- API endpoints

##### Technical Details

- Image input dimensions
- Vocabulary size
- Embedding dimensions
- Model checkpoint locations
- API endpoints and methods

## Diagram (Mermaid)

```mermaid
graph TB
    subgraph Data_Input
        ImageData["Flickr8k Image Dataset"]:::data
        TextData["Text Annotations"]:::data
        ExpertAnn["Expert Annotations"]:::data
        CrowdAnn["Crowd Annotations"]:::data
    end

    subgraph Data_Processing
        DataPrep["Data Preprocessing Pipeline"]:::processing
        TrainSplit["Training Data"]:::data
        TestSplit["Testing Data"]:::data
        DevSplit["Development Data"]:::data
        TokenProcess["Text Tokenization"]:::processing
        LemmaTokens["Lemmatized Tokens"]:::data
    end

    subgraph Model_Architecture
        ResNet["ResNet (CNN)"]:::model
        LSTM["LSTM Network"]:::model
        ModelImpl["Model Implementation"]:::model
    end

    subgraph Application_Layer
        WebApp["Flask Web Application"]:::app
    end

    %% Data Flow
    ImageData --> DataPrep
    TextData --> TokenProcess
    ExpertAnn --> TokenProcess
    CrowdAnn --> TokenProcess
    
    DataPrep --> TrainSplit
    DataPrep --> TestSplit
    DataPrep --> DevSplit
    
    TokenProcess --> LemmaTokens
    
    TrainSplit --> ModelImpl
    TestSplit --> ModelImpl
    DevSplit --> ModelImpl
    LemmaTokens --> ModelImpl
    
    ModelImpl --> ResNet
    ResNet --> LSTM
    LSTM --> WebApp

    %% Styling
    classDef data fill:#b3d9ff,stroke:#333,stroke-width:2px
    classDef processing fill:#90EE90,stroke:#333,stroke-width:2px
    classDef model fill:#ffb366,stroke:#333,stroke-width:2px
    classDef app fill:#d9b3ff,stroke:#333,stroke-width:2px

    %% Click Events
    click ImageData "https://github.com/Yash8745/Image-caption/tree/main/Flickr8k_Dataset/Flicker8k_Dataset"
    click TextData "https://github.com/Yash8745/Image-caption/blob/main/Flickr8k_text/Flickr8k.token.txt"
    click TrainSplit "https://github.com/Yash8745/Image-caption/blob/main/Flickr8k_text/Flickr_8k.trainImages.txt"
    click TestSplit "https://github.com/Yash8745/Image-caption/blob/main/Flickr8k_text/Flickr_8k.testImages.txt"
    click DevSplit "https://github.com/Yash8745/Image-caption/blob/main/Flickr8k_text/Flickr_8k.devImages.txt"
    click LemmaTokens "https://github.com/Yash8745/Image-caption/blob/main/Flickr8k_text/Flickr8k.lemma.token.txt"
    click ModelImpl "https://github.com/Yash8745/Image-caption/blob/main/Image Caption Generation using ResNet and LSTMs.ipynb"
    click WebApp "https://github.com/Yash8745/Image-caption/blob/main/app.py"
    click ExpertAnn "https://github.com/Yash8745/Image-caption/blob/main/Flickr8k_text/ExpertAnnotations.txt"
    click CrowdAnn "https://github.com/Yash8745/Image-caption/blob/main/Flickr8k_text/CrowdFlowerAnnotations.txt"
```
