# itorority0118/image-captioning architecture

> Architecture diagram of the GitHub repository itorority0118/image-captioning, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-11). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/itorority0118/image-captioning
- Repository: https://github.com/itorority0118/image-captioning

## Overview

This repository is an image-captioning research project comparing four image encoders and two decoder families on Flickr30k. The principal workflow prepares caption data and image splits, extracts reusable image features, trains a caption decoder, generates captions, and evaluates them. A separate documented Gradio demo is also a product capability, though its implementation is absent from the tree. The training excerpt uses precomputed features; evaluation and demo details are documented but unsampled, so their connections are left unverified.

Main entry points: [`src/train.py`](https://github.com/itorority0118/image-captioning/blob/main/src/train.py) (Training runner), [`app`](https://github.com/itorority0118/image-captioning/tree/main/app) (Gradio demo), [`src/data/dataset.py`](https://github.com/itorority0118/image-captioning/blob/main/src/data/dataset.py) (Caption dataset), [`src/data/text_preprocess.py`](https://github.com/itorority0118/image-captioning/blob/main/src/data/text_preprocess.py) (Caption preprocessing)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_data["Data preparation"]
  node_caption_cleaning["Caption preprocessing<br/>[text_preprocess.py]"]
  node_splits["Dataset splits<br/>[split.py]"]
  node_image_audit["Image exploration<br/>[explore_images.py]"]
  node_caption_dataset["Caption dataset<br/>[dataset.py]"]
end

subgraph group_features["Image features"]
  node_encoder_models["Image encoders<br/>[encoders.py]"]
  node_feature_extraction["Feature extraction<br/>[image_features.py]"]
  node_feature_store[("Feature files<br/>[paths.py]")]
end

subgraph group_training["Model training"]
  node_training_runner["Training runner<br/>[train.py]"]
  node_gpt2_decoder["GPT-2 decoder<br/>[decoder_gpt2.py]"]
  node_lstm_decoder["LSTM attention<br/>[decoder_lstm.py]"]
  node_checkpoint_store[("Checkpoints<br/>[paths.py]")]
end

subgraph group_generation["Caption generation"]
  node_greedy_search["Greedy decoding<br/>[greedy.py]"]
  node_beam_search["Beam decoding<br/>[beam.py]"]
  node_diverse_beam["Diverse beam<br/>[diverse_beam.py]"]
end

subgraph group_product["Evaluation and demo"]
  node_evaluation["Caption metrics"]
  node_web_demo["Gradio demo"]
end

node_researcher(("Researcher"))
node_flickr[("Flickr30k files")]
node_model_hub["Pretrained model hub"]

node_researcher -.->|"uses"| node_web_demo
node_flickr -.->|"captions and images"| node_caption_dataset
node_flickr -->|"audits"| node_image_audit
node_caption_dataset -->|"uses"| node_caption_cleaning
node_feature_extraction -->|"extracts with"| node_encoder_models
node_feature_extraction -->|"writes features"| node_feature_store
node_training_runner -->|"loads"| node_caption_dataset
node_training_runner -->|"locates features"| node_feature_store
node_training_runner -->|"constructs"| node_gpt2_decoder
node_training_runner -->|"constructs"| node_lstm_decoder
node_training_runner -->|"locates checkpoints"| node_checkpoint_store
node_greedy_search -->|"decodes tokens"| node_caption_cleaning
node_beam_search -->|"decodes tokens"| node_caption_cleaning
node_encoder_models -->|"loads pretrained weights"| node_model_hub
node_gpt2_decoder -->|"loads GPT-2"| node_model_hub
node_training_runner -.->|"evaluates"| node_evaluation
node_web_demo -.->|"generates captions"| node_greedy_search
node_web_demo -.->|"generates captions"| node_beam_search

click node_caption_cleaning "https://github.com/itorority0118/image-captioning/blob/main/src/data/text_preprocess.py"
click node_splits "https://github.com/itorority0118/image-captioning/blob/main/src/data/split.py"
click node_image_audit "https://github.com/itorority0118/image-captioning/blob/main/src/data/explore_images.py"
click node_caption_dataset "https://github.com/itorority0118/image-captioning/blob/main/src/data/dataset.py"
click node_encoder_models "https://github.com/itorority0118/image-captioning/blob/main/src/models/encoders.py"
click node_feature_extraction "https://github.com/itorority0118/image-captioning/blob/main/src/data/image_features.py"
click node_feature_store "https://github.com/itorority0118/image-captioning/blob/main/src/paths.py"
click node_training_runner "https://github.com/itorority0118/image-captioning/blob/main/src/train.py"
click node_gpt2_decoder "https://github.com/itorority0118/image-captioning/blob/main/src/models/decoder_gpt2.py"
click node_lstm_decoder "https://github.com/itorority0118/image-captioning/blob/main/src/models/decoder_lstm.py"
click node_checkpoint_store "https://github.com/itorority0118/image-captioning/blob/main/src/paths.py"
click node_greedy_search "https://github.com/itorority0118/image-captioning/blob/main/src/decoding/greedy.py"
click node_beam_search "https://github.com/itorority0118/image-captioning/blob/main/src/decoding/beam.py"
click node_diverse_beam "https://github.com/itorority0118/image-captioning/blob/main/src/decoding/diverse_beam.py"
click node_evaluation "https://github.com/itorority0118/image-captioning/tree/main/src/eval"
click node_web_demo "https://github.com/itorority0118/image-captioning/tree/main/app"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_caption_cleaning,node_splits,node_image_audit,node_caption_dataset toneBlue
class node_encoder_models,node_feature_extraction,node_feature_store,node_flickr toneAmber
class node_training_runner,node_gpt2_decoder,node_lstm_decoder,node_checkpoint_store toneMint
class node_greedy_search,node_beam_search,node_diverse_beam toneRose
class node_evaluation,node_web_demo,node_researcher,node_model_hub toneIndigo
```

## Components

### Data preparation

- [Caption preprocessing](https://github.com/itorority0118/image-captioning/blob/main/src/data/text_preprocess.py): `src/data/text_preprocess.py`
- [Dataset splits](https://github.com/itorority0118/image-captioning/blob/main/src/data/split.py): `src/data/split.py`
- [Image exploration](https://github.com/itorority0118/image-captioning/blob/main/src/data/explore_images.py): `src/data/explore_images.py`
- [Caption dataset](https://github.com/itorority0118/image-captioning/blob/main/src/data/dataset.py): `src/data/dataset.py`

### Image features

- [Image encoders](https://github.com/itorority0118/image-captioning/blob/main/src/models/encoders.py): `src/models/encoders.py`
- [Feature extraction](https://github.com/itorority0118/image-captioning/blob/main/src/data/image_features.py): `src/data/image_features.py`
- [Feature files](https://github.com/itorority0118/image-captioning/blob/main/src/paths.py): `src/paths.py`

### Model training

- [Training runner](https://github.com/itorority0118/image-captioning/blob/main/src/train.py): `src/train.py`
- [GPT-2 decoder](https://github.com/itorority0118/image-captioning/blob/main/src/models/decoder_gpt2.py): `src/models/decoder_gpt2.py`
- [LSTM attention](https://github.com/itorority0118/image-captioning/blob/main/src/models/decoder_lstm.py): `src/models/decoder_lstm.py`
- [Checkpoints](https://github.com/itorority0118/image-captioning/blob/main/src/paths.py): `src/paths.py`

### Caption generation

- [Greedy decoding](https://github.com/itorority0118/image-captioning/blob/main/src/decoding/greedy.py): `src/decoding/greedy.py`
- [Beam decoding](https://github.com/itorority0118/image-captioning/blob/main/src/decoding/beam.py): `src/decoding/beam.py`
- [Diverse beam](https://github.com/itorority0118/image-captioning/blob/main/src/decoding/diverse_beam.py): `src/decoding/diverse_beam.py`

### Evaluation and demo

- [Caption metrics](https://github.com/itorority0118/image-captioning/tree/main/src/eval): `src/eval`
- [Gradio demo](https://github.com/itorority0118/image-captioning/tree/main/app): `app`

### Other components

- **Researcher**
- **Flickr30k files**
- **Pretrained model hub**

## Connections

- Researcher → Gradio demo: uses (evidence: `README.md`)
- Flickr30k files → Caption dataset: captions and images (evidence: `README.md`)
- Flickr30k files → Image exploration: audits (evidence: `src/data/explore_images.py`)
- Caption dataset → Caption preprocessing: uses (evidence: `src/data/dataset.py`)
- Feature extraction → Image encoders: extracts with (evidence: `src/data/image_features.py`)
- Feature extraction → Feature files: writes features (evidence: `src/data/image_features.py`)
- Training runner → Caption dataset: loads (evidence: `src/train.py`)
- Training runner → Feature files: locates features (evidence: `src/train.py`)
- Training runner → GPT-2 decoder: constructs (evidence: `src/train.py`)
- Training runner → LSTM attention: constructs (evidence: `src/train.py`)
- Training runner → Checkpoints: locates checkpoints (evidence: `src/train.py`)
- Greedy decoding → Caption preprocessing: decodes tokens (evidence: `src/decoding/greedy.py`)
- Beam decoding → Caption preprocessing: decodes tokens (evidence: `src/decoding/beam.py`)
- Image encoders → Pretrained model hub: loads pretrained weights (evidence: `src/models/encoders.py`)
- GPT-2 decoder → Pretrained model hub: loads GPT-2 (evidence: `src/models/decoder_gpt2.py`)
- Training runner → Caption metrics: evaluates
- Gradio demo → Greedy decoding: generates captions
- Gradio demo → Beam decoding: generates captions