# vision-cair/visualgpt architecture

> Architecture diagram of the GitHub repository vision-cair/visualgpt, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-04). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/vision-cair/visualgpt
- Repository: https://github.com/vision-cair/visualgpt

## Overview

##### Project type and purpose
- VisualGPT is a research‐code image captioning pipeline that adapts a pretrained GPT-2 language model with visual features (COCO detections) for data-efficient caption generation.
- It isn’t a client-server webapp or microservices system; rather it’s an ML training and evaluation workflow.

##### File structure
- Top-level files:
- train_visualGPT.py – orchestrates training
- environment.yml, requirements.txt – dependencies
- data/ – modules for dataset handling, tokenization, BPE vocab, I/O utilities
- models/ –
- transformer/ – attention, encoders, GPT2 loader, core VisualGPT model
- beam_search/ – beam search inference logic
- captioning_model.py, containers.py – higher-level model interfaces
- evaluation/ – BLEU, METEOR, CIDEr, ROUGE scorers and tokenizer wrapper
- utils/ – type definitions, common utilities

##### README
- Confirms use of GPT-2 pretrained weights, COCO features HDF5, spacy, conda env
- Highlights modules from HuggingFace Transformers and Meshed Memory Transformer
- Shows a final architecture image (images/final_architecture.jpg)

##### Components
a. Data Ingestion & Preparation
- Raw COCO annotations (JSON)
- coco_detections.hdf5 (visual feature store)
- data/custom_readers, tokenization modules, vocab.bpe
b. Pretrained Modules & Encoders
- GPT-2 model loader (models/transformer/load_gptmodel.py)
- Visual feature encoder (no CNN here—features precomputed)
c. VisualGPT Model
- Transformer-based fusion module (models/transformer/transformer_visualGPT.py)
- Configuration (models/transformer/config.py)
- Attention, encoders (models/transformer/attention.py, encoders.py)
d. Inference & Decoding
- Beam search (models/beam_search/beam_search.py)
- Greedy sampling (optional)
e. Training Pipeline
- train_visualGPT.py – argument parser, optimizer (AdamW), gradient accumulation, logging
- containers.py – model checkpointing
f. Evaluation
- evaluation/tokenizer.py – text preprocessing
- evaluation/bleu, cider, meteor, rouge modules
g. External Dependencies
- GPT-2 weights from HuggingFace S3
- spacy English model
- Conda environment

##### Relationships & Data Flow
- Control flow: train_visualGPT.py orchestrates epochs → optimizer → model updates → logging.

##### Architectural patterns & design principles
- Modular separation: data, model, evaluation, utils
- Reuse of pretrained components (GPT-2), separation of feature extraction from training
- Clear layering: Data → Model → Decoding → Evaluation → Logging

##### Essential elements checklist
- Component names & file paths
- Clear boundaries for major subsystems (Data Prep, Model, Evaluation)
- Notes on external downloads (GPT-2 weights, spacy model)

## Diagram (Mermaid)

```mermaid
flowchart TD
    subgraph "Data Preparation"
        COCO["COCO JSON Annotations"]:::dataSource
        HDF5["coco_detections.hdf5"]:::dataSource
        Readers["custom_readers.py"]:::process
        Datasets["custom_datasets.py"]:::process
        FileUtils["file_utils.py"]:::process
        TokGPT2["tokenization_gpt2.py"]:::process
        TokUtils["tokenization_utils.py"]:::process
        Vocab["vocab.bpe"]:::dataStore
        EncoderBPE["encoder.json"]:::dataStore
    end

    subgraph "Pretrained Modules & Encoders"
        GPT2Loader["load_gptmodel.py"]:::process
    end

    subgraph "VisualGPT Model"
        Fusion["transformer_visualGPT.py"]:::process
        Config["config.py"]:::process
        Attention["attention.py"]:::process
        Encoders["encoders.py"]:::process
        DecoderVis["gpt_decoder_visualGPT.py"]:::process
    end

    subgraph "Inference & Decoding"
        Beam["beam_search.py"]:::process
        SamplingUtils["utils.py"]:::process
    end

    subgraph "Training Pipeline"
        Train["train_visualGPT.py"]:::process
        Containers["containers.py"]:::process
        AdamW["Optimizer (AdamW)"]:::external
    end

    subgraph "Evaluation"
        TokenEval["tokenizer.py"]:::eval
        BLEU["bleu.py"]:::eval
        BLEU_Scorer["bleu_scorer.py"]:::eval
        CIDEr["cider.py"]:::eval
        CIDEr_S["cider_scorer.py"]:::eval
        METEOR["meteor.py"]:::eval
        ROUGE["rouge.py"]:::eval
    end

    subgraph "External Dependencies"
        EnvYml["environment.yml"]:::external
        Req["requirements.txt"]:::external
        Readme["README.md"]:::external
        GPT2Ext["GPT-2 Weights (HuggingFace S3)"]:::dataStore
        Spacy["Spacy English Model"]:::dataStore
    end

    COCO --> Readers
    HDF5 --> Readers
    COCO --> Datasets
    HDF5 --> Datasets
    Readers --> DataBatch["Batched visual+text input"]:::process
    Datasets --> DataBatch
    FileUtils --> DataBatch
    TokGPT2 --> DataBatch
    TokUtils --> DataBatch
    Vocab --> DataBatch
    EncoderBPE --> DataBatch

    DataBatch --> Fusion
    GPT2Loader --> Fusion
    Config --> Fusion
    Attention --> Fusion
    Encoders --> Fusion
    DecoderVis --> Fusion

    Fusion --> Beam
    Fusion --> SamplingUtils
    Beam --> Generated["Generated Captions"]:::process
    SamplingUtils --> Generated

    Generated --> TokenEval
    TokenEval --> BLEU
    TokenEval --> BLEU_Scorer
    TokenEval --> CIDEr
    TokenEval --> CIDEr_S
    TokenEval --> METEOR
    TokenEval --> ROUGE

    Train --> Fusion
    Train --> Containers
    Train --> AdamW
    AdamW -.-> Fusion
    Train -.-> DataBatch

    EnvYml --> Readme
    Req --> Readme
    GPT2Ext --> GPT2Loader
    Spacy --> TokenEval

    classDef dataSource fill:#c8e6c9,stroke:#388e3c
    classDef dataStore fill:#bbdefb,stroke:#1976d2,stroke-width:2px
    classDef process fill:#ffe0b2,stroke:#ef6c00
    classDef eval fill:#e1bee7,stroke:#8e24aa
    classDef external fill:#f0f4c3,stroke:#afb42b

    click Readers "https://github.com/vision-cair/visualgpt/blob/main/data/custom_readers.py"
    click Datasets "https://github.com/vision-cair/visualgpt/blob/main/data/custom_datasets.py"
    click FileUtils "https://github.com/vision-cair/visualgpt/blob/main/data/file_utils.py"
    click TokGPT2 "https://github.com/vision-cair/visualgpt/blob/main/data/tokenization_gpt2.py"
    click TokUtils "https://github.com/vision-cair/visualgpt/blob/main/data/tokenization_utils.py"
    click Vocab "https://github.com/vision-cair/visualgpt/blob/main/data/vocab.bpe"
    click EncoderBPE "https://github.com/vision-cair/visualgpt/blob/main/data/encoder.json"
    click GPT2Loader "https://github.com/vision-cair/visualgpt/blob/main/models/transformer/load_gptmodel.py"
    click Fusion "https://github.com/vision-cair/visualgpt/blob/main/models/transformer/transformer_visualGPT.py"
    click Config "https://github.com/vision-cair/visualgpt/blob/main/models/transformer/config.py"
    click Attention "https://github.com/vision-cair/visualgpt/blob/main/models/transformer/attention.py"
    click Encoders "https://github.com/vision-cair/visualgpt/blob/main/models/transformer/encoders.py"
    click DecoderVis "https://github.com/vision-cair/visualgpt/blob/main/models/transformer/gpt_decoder_visualGPT.py"
    click Beam "https://github.com/vision-cair/visualgpt/blob/main/models/beam_search/beam_search.py"
    click SamplingUtils "https://github.com/vision-cair/visualgpt/blob/main/models/transformer/utils.py"
    click Train "https://github.com/vision-cair/visualgpt/blob/main/train_visualGPT.py"
    click Containers "https://github.com/vision-cair/visualgpt/blob/main/models/containers.py"
    click TokenEval "https://github.com/vision-cair/visualgpt/blob/main/evaluation/tokenizer.py"
    click BLEU "https://github.com/vision-cair/visualgpt/blob/main/evaluation/bleu/bleu.py"
    click BLEU_Scorer "https://github.com/vision-cair/visualgpt/blob/main/evaluation/bleu/bleu_scorer.py"
    click CIDEr "https://github.com/vision-cair/visualgpt/blob/main/evaluation/cider/cider.py"
    click CIDEr_S "https://github.com/vision-cair/visualgpt/blob/main/evaluation/cider/cider_scorer.py"
    click METEOR "https://github.com/vision-cair/visualgpt/blob/main/evaluation/meteor/meteor.py"
    click ROUGE "https://github.com/vision-cair/visualgpt/blob/main/evaluation/rouge/rouge.py"
    click EnvYml "https://github.com/vision-cair/visualgpt/blob/main/environment.yml"
    click Req "https://github.com/vision-cair/visualgpt/blob/main/requirements.txt"
    click Readme "https://github.com/vision-cair/visualgpt/blob/main/README.md"
```
