# abdelkadirsellahi/vqa-using-yolo-bert-gpt-2 architecture

> Architecture diagram of the GitHub repository abdelkadirsellahi/vqa-using-yolo-bert-gpt-2, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2024-12-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/abdelkadirsellahi/vqa-using-yolo-bert-gpt-2
- Repository: https://github.com/abdelkadirsellahi/vqa-using-yolo-bert-gpt-2

## Overview

##### Component Details

For each major component, create sub-components showing the internal processing:

a) YOLOv8 Block:
- Three output branches:
- Confidence Scores
- Class Values
- A "Feature Extraction" sub-block

b) BERT Block:
- Processing steps:
- Tokenization
- Input IDs
- Attention Masks
- Text Embeddings

c) Feature Fusion Block:
- Combination of:
- Visual Features
- Text Embeddings
- Linear Projection Layer

d) GPT-2 Block:
- Processing of:
- Combined Features Input
- Logits Generation
- Answer Selection

- Blue for image-related data
- Green for text-related data
- Purple for combined features

##### Additional Elements

a) Processing Details:
- Image size (224x224)
- Batch size (64)
- Hidden dimension (256)

b) Model Parameters:
- Learning rate (1e-5)
- Number of epochs (30)

##### Technical Details
Annotations for:
- Input/Output formats
- Key transformations
- Model architectures
- Performance metrics

##### Optimization
- Maintaining proper spacing
- Grouping related components
- Using clear, readable labels

- Training (VQA_YOLO_BERT_GPT2_Training.ipynb)
- Validation (VQA_YOLO_BERT_GPT2_Validation.ipynb)
- Prediction (VQA_YOLO_BERT_GPT2_Prediction.ipynb)

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Define styles
    classDef input fill:#a8d5ff,stroke:#333,stroke-width:2px
    classDef model fill:#ffcca8,stroke:#333,stroke-width:2px
    classDef process fill:#d5ffa8,stroke:#333,stroke-width:2px
    classDef output fill:#ffa8d5,stroke:#333,stroke-width:2px
    classDef pipeline fill:#f0f0f0,stroke:#333,stroke-width:2px

    %% Input Layer
    I[Image Input]:::input
    Q[Question Input]:::input
    
    %% Processing Layer
    subgraph YOLO[YOLOv8 Processing]
        Y1[Feature Extraction]
        Y2[Bounding Boxes]
        Y3[Confidence Scores]
        Y4[Class Values]
    end

    subgraph BERT[BERT Processing]
        B1[Tokenization]
        B2[Input IDs]
        B3[Attention Masks]
        B4[Text Embeddings]
    end

    subgraph Fusion[Feature Fusion]
        F1[Visual Features]
        F2[Text Features]
        F3[Linear Projection]
    end

    subgraph GPT[GPT-2 Processing]
        G1[Combined Features]
        G2[Logits Generation]
        G3[Answer Selection]
    end

    %% Output Layer
    A[Answer Generation]:::output

    %% Pipeline Sections
    subgraph Training[Training Pipeline]
        T1[Loss Calculation]
        T2[Model Updates]
    end

    subgraph Validation[Validation Pipeline]
        V1[Metrics Evaluation]
        V2[Performance Analysis]
    end

    subgraph Prediction[Prediction Pipeline]
        P1[Real-time Processing]
        P2[Answer Generation]
    end

    %% Connections
    I --> Y1
    Q --> B1
    B1 --> B2 --> B3 --> B4
    Y1 --> Y2 & Y3 & Y4
    Y2 & Y3 & Y4 --> F1
    B4 --> F2
    F1 & F2 --> F3
    F3 --> G1 --> G2 --> G3
    G3 --> A

    %% Click events for component mapping
    click Training "https://github.com/AbdelkadirSellahi/VQA-Using-YOLO-BERT-GPT-2/blob/main/VQA_YOLO_BERT_GPT2_Training.ipynb"
    click Validation "https://github.com/AbdelkadirSellahi/VQA-Using-YOLO-BERT-GPT-2/blob/main/VQA_YOLO_BERT_GPT2_Validation.ipynb"
    click Prediction "https://github.com/AbdelkadirSellahi/VQA-Using-YOLO-BERT-GPT-2/blob/main/VQA_YOLO_BERT_GPT2_Prediction.ipynb"
    click I "https://github.com/AbdelkadirSellahi/VQA-Using-YOLO-BERT-GPT-2/tree/main/img/"

    %% Legend
    subgraph Legend
        L1[Input]:::input
        L2[Model Component]:::model
        L3[Process]:::process
        L4[Output]:::output
    end
```
