How codezakh/exploiting-bert-thru-translation works
Project type and purpose
• This is an open-source research tool for multimodal target sentiment classification.
• It fine-tunes BERT on tweets (text) augmented with image captions generated by the CATR captioning model.
• It implements two fusion strategies: Early Fusion (EF_CapBERT) and Late Fusion (LF_CapBERT), for the Twitter-15 and Twitter-17 datasets.
File structure
Top-level files:
• EF_CapBERT_Tw15.py, EF_CapBERT_Tw17.py, LF_CapBERT.py – training/evaluation entrypoints for each fusion method and dataset.
• caption_multiple.py – helper to batch-generate captions via the CATR submodule.
• requirements.in – Python dependencies.
• captions/ – directory containing pre-generated JSON caption files for both datasets.
• catr/ – Git submodule housing the CATR image captioning model.
• README.md – high-level description and instructions.
README for additional insights
• Describes input data (TSV files for train/dev/test, captions JSON).
• Explains how to generate captions by cloning submodule and running caption_multiple.py.
• Shows external links to datasets and CATR repo, plus architecture image in figures/.
Components
a. Data Sources
• Twitter-15 & Twitter-17 TSV datasets (train/dev/test splits).
• captions/twitter15_images.json & twitter17_images.json.
b. Preprocessing & Caption Generation
• CATR module (submodule under catr/) invoked by caption_multiple.py to generate JSON captions from raw images.
c. Model Training & Fusion Strategies
• Early Fusion pipeline (EF_CapBERT_Tw15.py / EF_CapBERT_Tw17.py)
– Input: raw text tokens + caption tokens → concatenation → BERT encoder → classification head.
• Late Fusion pipeline (LF_CapBERT.py)
– Input: text tokens → BERT encoder → text embedding
caption tokens → BERT encoder → caption embedding
→ merge embeddings (e.g., concatenation or weighted sum) → classification head.
d. Infrastructure & Tooling
• Python environment managed by requirements.in
• External services: none at runtime, only local usage of CATR model.
• Output: evaluation metrics, model checkpoints, logs.
Relationships and interactions
• Data files feed into preprocessing or directly into model scripts.
• caption_multiple.py reads raw images (via CATR) → writes captions JSON → used by EF/LF scripts.
• EF/LF scripts import BERT (via HuggingFace Transformers) and load data + captions → train & evaluate.
Architectural patterns & design principles
• Modular separation of concerns:
• Data ingestion and preprocessing (caption generation)
• Model definition and fusion logic
• Training/evaluation orchestration
• Reusability: caption_multiple.py can be reused for other captioning tasks.
• Extensibility: adding new fusion strategies or datasets involves adding new entrypoint scripts.
Relevant technologies, frameworks, libraries
• Python 3.x
• PyTorch (assumed by CATR & BERT fine-tuning)
• HuggingFace Transformers (BERT)
• CATR image captioning model (submodule)
• JSON, TSV for data interchange
Diagramming recommendations
• Raw images → CATR → captions JSON
• TSV + JSON → Fusion script → BERT → outputs