# codezakh/exploiting-bert-thru-translation architecture

> Architecture diagram of the GitHub repository codezakh/exploiting-bert-thru-translation, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-16). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/codezakh/exploiting-bert-thru-translation
- Repository: https://github.com/codezakh/exploiting-bert-thru-translation

## Overview

##### Project type and purpose
- This is an open-source research tool for multimodal target sentiment classification.
- It fine-tunes BERT on tweets (text) augmented with image captions generated by the CATR captioning model.
- It implements two fusion strategies: Early Fusion (EF_CapBERT) and Late Fusion (LF_CapBERT), for the Twitter-15 and Twitter-17 datasets.

##### File structure
Top-level files:
- EF_CapBERT_Tw15.py, EF_CapBERT_Tw17.py, LF_CapBERT.py – training/evaluation entrypoints for each fusion method and dataset.
- caption_multiple.py – helper to batch-generate captions via the CATR submodule.
- requirements.in – Python dependencies.
- captions/ – directory containing pre-generated JSON caption files for both datasets.
- catr/ – Git submodule housing the CATR image captioning model.
- README.md – high-level description and instructions.

##### README for additional insights
- Describes input data (TSV files for train/dev/test, captions JSON).
- Explains how to generate captions by cloning submodule and running caption_multiple.py.
- Shows external links to datasets and CATR repo, plus architecture image in figures/.

##### Components
a. Data Sources
- Twitter-15 & Twitter-17 TSV datasets (train/dev/test splits).
- captions/twitter15_images.json & twitter17_images.json.
b. Preprocessing & Caption Generation
- CATR module (submodule under catr/) invoked by caption_multiple.py to generate JSON captions from raw images.
c. Model Training & Fusion Strategies
- Early Fusion pipeline (EF_CapBERT_Tw15.py / EF_CapBERT_Tw17.py)
– Input: raw text tokens + caption tokens → concatenation → BERT encoder → classification head.
- Late Fusion pipeline (LF_CapBERT.py)
– Input: text tokens → BERT encoder → text embedding
caption tokens → BERT encoder → caption embedding
→ merge embeddings (e.g., concatenation or weighted sum) → classification head.
d. Infrastructure & Tooling
- Python environment managed by requirements.in
- External services: none at runtime, only local usage of CATR model.
- Output: evaluation metrics, model checkpoints, logs.

##### Relationships and interactions
- Data files feed into preprocessing or directly into model scripts.
- caption_multiple.py reads raw images (via CATR) → writes captions JSON → used by EF/LF scripts.
- EF/LF scripts import BERT (via HuggingFace Transformers) and load data + captions → train & evaluate.

##### Architectural patterns & design principles
- Modular separation of concerns:
- Data ingestion and preprocessing (caption generation)
- Model definition and fusion logic
- Training/evaluation orchestration
- Reusability: caption_multiple.py can be reused for other captioning tasks.
- Extensibility: adding new fusion strategies or datasets involves adding new entrypoint scripts.

##### Relevant technologies, frameworks, libraries
- Python 3.x
- PyTorch (assumed by CATR & BERT fine-tuning)
- HuggingFace Transformers (BERT)
- CATR image captioning model (submodule)
- JSON, TSV for data interchange

##### Diagramming recommendations
- Raw images → CATR → captions JSON
- TSV + JSON → Fusion script → BERT → outputs

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Data Sources
    subgraph "Data Sources"
        direction TB
        Twitter15TSV["Twitter-15 TSV Splits"]:::dataStore
        Twitter17TSV["Twitter-17 TSV Splits"]:::dataStore
        Twitter15JSON["captions/twitter2015_images.json"]:::dataStore
        Twitter17JSON["captions/twitter2017_images.json"]:::dataStore
    end

    %% Preprocessing & Caption Generation
    subgraph "Preprocessing & Caption Generation"
        direction TB
        CaptionGen["caption_multiple.py"]:::prep
        CATR["CATR Submodule (catr/)"]:::prep
    end

    %% Model Training & Fusion
    subgraph "Model Training & Fusion"
        direction TB

        subgraph "Early Fusion"
            direction TB
            EFC15["EF_CapBERT_Tw15.py"]:::script
            EFC17["EF_CapBERT_Tw17.py"]:::script
            EF_Input["Text Tokens + Caption Tokens"]:::model
            EF_BERT["BERT Encoder"]:::model
            EF_Head["Classification Head"]:::model
        end

        subgraph "Late Fusion"
            direction TB
            LFC["LF_CapBERT.py"]:::script
            TextEnc["Text BERT Encoder"]:::model
            CapEnc["Caption BERT Encoder"]:::model
            Merge["Merge Embeddings"]:::model
            LF_Head["Classification Head"]:::model
        end
    end

    %% Infrastructure & Documentation
    subgraph "Infrastructure & Documentation"
        direction TB
        Requirements["requirements.in"]:::script
        README["README.md"]:::script
        ArchImg["figures/archdiag.png"]:::script
    end

    %% Connections
    Twitter15TSV --> CaptionGen
    Twitter17TSV --> CaptionGen
    CaptionGen -->|"generates captions JSON"| Twitter15JSON
    CaptionGen --> Twitter17JSON

    Twitter15TSV --> EFC15
    Twitter15JSON --> EFC15
    EFC15 --> EF_Input
    EF_Input --> EF_BERT
    EF_BERT --> EF_Head

    Twitter17TSV --> EFC17
    Twitter17JSON --> EFC17
    EFC17 --> EF_Input

    Twitter15TSV --> LFC
    Twitter15JSON --> LFC
    LFC --> TextEnc
    Twitter15JSON --> LFC
    LFC --> CapEnc
    TextEnc --> Merge
    CapEnc --> Merge
    Merge --> LF_Head

    Requirements --> CaptionGen
    Requirements --> EFC15
    Requirements --> EFC17
    Requirements --> LFC

    README --> CaptionGen
    README --> EFC15
    README --> EFC17
    README --> LFC
    README --> ArchImg

    %% Click Events
    click Twitter15JSON "https://github.com/codezakh/exploiting-bert-thru-translation/blob/master/captions/twitter2015_images.json"
    click Twitter17JSON "https://github.com/codezakh/exploiting-bert-thru-translation/blob/master/captions/twitter2017_images.json"
    click CaptionGen "https://github.com/codezakh/exploiting-bert-thru-translation/blob/master/caption_multiple.py"
    click CATR "https://github.com/codezakh/exploiting-bert-thru-translation/tree/master/catr/"
    click EFC15 "https://github.com/codezakh/exploiting-bert-thru-translation/blob/master/EF_CapBERT_Tw15.py"
    click EFC17 "https://github.com/codezakh/exploiting-bert-thru-translation/blob/master/EF_CapBERT_Tw17.py"
    click LFC "https://github.com/codezakh/exploiting-bert-thru-translation/blob/master/LF_CapBERT.py"
    click Requirements "https://github.com/codezakh/exploiting-bert-thru-translation/blob/master/requirements.in"
    click README "https://github.com/codezakh/exploiting-bert-thru-translation/blob/master/README.md"
    click ArchImg "https://github.com/codezakh/exploiting-bert-thru-translation/blob/master/figures/archdiag.png"

    %% Styles
    classDef dataStore fill:#B3CDE3,stroke:#1F78B4,color:#000
    classDef prep fill:#C7E9C0,stroke:#33A02C,color:#000
    classDef model fill:#FEC44F,stroke:#E31A1C,color:#000
    classDef script fill:#D9D9D9,stroke:#636363,color:#000
```
