# vladman-25/image-captioning-experiments architecture

> Architecture diagram of the GitHub repository vladman-25/image-captioning-experiments, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2024-12-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/vladman-25/image-captioning-experiments
- Repository: https://github.com/vladman-25/image-captioning-experiments

## Overview

##### Main Components

A. Data Pipeline
- Input Layer (Images from Flickr30k dataset)
- Text Processing Layer (Romanian language captions)
- Word Embedding Components:
- Custom Embeddings (FastText, Word2Vec, RoBERT)
- Multiple embedding variations (CC, CoRoLa, NLPL)

B. Model Architecture
- Feature Extraction:
- EfficientNetB4 (as indicated by weights folder)
- Caption Generation Models:
- RNN-based architecture
- Transformer-based architecture
- Multiple Model Variants (as shown by different notebooks)

C. Inference Pipeline
- Dedicated inference component (shown by Inference notebook)

**[Input Layer]**
- ↓
**[Feature Extraction: EfficientNetB4]**
- ↓
**[Embedding Layer]**
- ↙     ↓     ↘
**[FastText] [Word2Vec] [RoBERT]**
- ↘     ↓     ↙
**[Caption Generation Models]**
- ↙           ↘
- [RNN]    [Transformer]
- ↘           ↙
**[Inference Engine]**
- ↓
**[Output: Romanian Captions]**

##### Important Details to Highlight

A. Multiple Embedding Paths:
- Parallel paths for different embedding types
- Which embedding is used with which architecture

B. Model Variants:
- Clearly separate RNN and Transformer approaches
- How they connect to the embedding layers

C. Training vs. Inference:

##### Additional Elements

- Different line types (training vs. inference)
- Embedding types

B. Add annotations for:
- Input image specifications
- Output caption format
- Model checkpoint locations

##### Technical Details to Note
- EfficientNetB4 weights location
- Different notebook implementations
- Dataset connection to Flickr30k-Romanian

##### Implementation Notes
- Use standardized UML or similar notation
- Keep components proportional to their importance
- Version information where relevant
- Clear data flow directions

## Diagram (Mermaid)

```mermaid
flowchart TB
    subgraph Input
        I[Input Images]
        C[Romanian Captions]
    end

    subgraph "Feature Extraction"
        FE["EfficientNetB4"]:::featureExtraction
    end

    subgraph "Embedding Layer"
        direction TB
        CC["CC FastText"]:::embedding
        CR["CoRoLa FastText"]:::embedding
        WV["NLPL Word2Vec"]:::embedding
        RB["RoBERT"]:::embedding
    end

    subgraph "Model Architecture"
        direction TB
        RNN["RNN Model"]:::model
        TR["Transformer Model"]:::model
    end

    subgraph "Inference"
        INF["Inference Engine"]:::inference
    end

    O["Romanian Caption Output"]:::output

    %% Connections
    I --> FE
    C --> CC & CR & WV & RB
    FE --> CC & CR & WV & RB
    CC & CR & WV & RB --> RNN & TR
    RNN & TR --> INF
    INF --> O

    %% Styles
    classDef input fill:#a8d1ff
    classDef featureExtraction fill:#90EE90
    classDef embedding fill:#FFE5B4
    classDef model fill:#FFA07A
    classDef inference fill:#DDA0DD
    classDef output fill:#a8d1ff

    %% Click Events
    click FE "https://github.com/vladman-25/Image-Captioning-Experiments/blob/master/weights/EfficientNetB4-v4-10.tf"
    click CC "https://github.com/vladman-25/Image-Captioning-Experiments/blob/master/CustomEmbeddings/Image_Captioning_Framework_CC_FastText.ipynb"
    click CR "https://github.com/vladman-25/Image-Captioning-Experiments/blob/master/CustomEmbeddings/Image_Captioning_Framework_CoRoLa_FastText.ipynb"
    click WV "https://github.com/vladman-25/Image-Captioning-Experiments/blob/master/CustomEmbeddings/Image_Captioning_Framework_NLPL_Word2Vec.ipynb"
    click RB "https://github.com/vladman-25/Image-Captioning-Experiments/blob/master/CustomEmbeddings/Image_Captioning_Framework_RoBert.ipynb"
    click RNN "https://github.com/vladman-25/Image-Captioning-Experiments/blob/master/Image_Captioning_Framework_RNN.ipynb"
    click TR "https://github.com/vladman-25/Image-Captioning-Experiments/blob/master/Image_Captioning_Framework_Transformers.ipynb"
    click INF "https://github.com/vladman-25/Image-Captioning-Experiments/blob/master/Image_Captioning_Framework_Inference.ipynb"

    %% Legend
    subgraph Legend
        L1["Input/Output"]:::input
        L2["Feature Extraction"]:::featureExtraction
        L3["Embedding Layer"]:::embedding
        L4["Model Architecture"]:::model
        L5["Inference"]:::inference
    end
```
