How jiaxuan-li/evcap works
Project type and purpose
• This is a research‐focused full‐stack system combining:
– A retrieval‐augmented image‐captioning model (“EVCap”) that integrates external visual‐name memory with vision and language modules.
– A lightweight web demo for running inference (index.html + static assets + search.py).
– Training and evaluation pipelines for benchmarking on COCO, NoCaps, Flickr30k.
File structure
• Top‐level scripts:
– train_evcap.py: training entry point
– eval_evcap.py & scripts/eval_evcap_*.sh: evaluation drivers
– search.py: inference web‐service backend
– search or static/index.html + static/js/index.js: client UI
• Directories:
– common/: utilities (config, logging, registry, distributed utils, Grad‐CAM)
– dataset/: COCO dataset loader
– ext_data/: prebuilt external memory (visual embeddings + names)
– models/: core modules (Vision encoders: eva_vit, blip2; QFormer; LLaMA adapter; evcap model wrapper)
– prompts/: prompt templates for LLM input
• Config files: environment.yaml, optims.py, common/config.py
• Static assets: Bulma CSS/JS for the demo front‐end
Extract architecture insights from the README
• EVCap interleaves:
1. Vision encoding (e.g., BLIP2 or EVA‐ViT) →
2. Query‐former (QFormer) to project vision features →
3. Memory retrieval (external visual‐name store) to fetch relevant object names →
4. Prompt assembly (prompts/prompt_evcap.txt) →
5. Language model (LLaMA) to generate captions.
• External memory is a pluggable module that can be updated without retraining the main model.
– Front-end UI (Browser)
– Inference API Server (search.py)
– Vision Encoder module (models/blip2.py, eva_vit)
– Q-Former module (models/Qformer.py)
– External Memory Store (ext_data/ext_memory_lvis.pkl + retrieval engine in common/dist_utils.py or search.py)
– Prompt Constructor (prompts/prompt_evcap.txt + common/utils.py)
– Language Model (models/modeling_llama.py)
– Training Pipeline (train_evcap.py, dataset/coco_dataset.py, optims.py)
– Evaluation Pipeline (scripts/eval_evcap_*.sh, eval_evcap.py)
– Common Utilities & Configuration (common/)
– Browser → HTTP POST image → Inference API Server
– Server splits path:
• Vision → Vision Encoder → Q-Former → query embeddings → Memory Store lookup → retrieved names
• Assembled prompt = vision tokens + retrieved names → fed into LLM → text caption
– Caption → API response → Browser display
– Training Pipeline: raw images + annotations → DataLoader → Model (Vision + Q-Former + LLM adapter) → loss backprop (common/optims.py)
– Evaluation Pipeline: loads trained checkpoint → runs inference on benchmark sets → outputs metrics
c. Architectural patterns & design principles:
– Modularity: separate vision, retrieval, prompting, language components
– Pluggable memory: external store decoupled from core model
– Registry/Factory (common/registry.py) for dynamic component instantiation
– Config‐driven (common/config.py + environment.yaml) for reproducibility
d. Key technologies & frameworks:
– Python 3, PyTorch for modeling
– Bulma CSS & JavaScript for front-end
– Memory retrieval (e.g., FAISS or precomputed vectors via common/dist_utils.py)
– LLaMA weights (Hugging Face or local) interfaced via modeling_llama.py