# jiaxuan-li/evcap architecture

> Architecture diagram of the GitHub repository jiaxuan-li/evcap, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-13). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/jiaxuan-li/evcap
- Repository: https://github.com/jiaxuan-li/evcap

## Overview

##### Project type and purpose
- This is a research‐focused full‐stack system combining:
– A retrieval‐augmented image‐captioning model (“EVCap”) that integrates external visual‐name memory with vision and language modules.
– A lightweight web demo for running inference (index.html + static assets + search.py).
– Training and evaluation pipelines for benchmarking on COCO, NoCaps, Flickr30k.

##### File structure
- Top‐level scripts:
– train_evcap.py: training entry point
– eval_evcap.py & scripts/eval_evcap_*.sh: evaluation drivers
– search.py: inference web‐service backend
– search or static/index.html + static/js/index.js: client UI
- Directories:
– common/: utilities (config, logging, registry, distributed utils, Grad‐CAM)
– dataset/: COCO dataset loader
– ext_data/: prebuilt external memory (visual embeddings + names)
– models/: core modules (Vision encoders: eva_vit, blip2; QFormer; LLaMA adapter; evcap model wrapper)
– prompts/: prompt templates for LLM input
- Config files: environment.yaml, optims.py, common/config.py
- Static assets: Bulma CSS/JS for the demo front‐end

##### Extract architecture insights from the README
- EVCap interleaves:
1. Vision encoding (e.g., BLIP2 or EVA‐ViT) →
2. Query‐former (QFormer) to project vision features →
3. Memory retrieval (external visual‐name store) to fetch relevant object names →
4. Prompt assembly (prompts/prompt_evcap.txt) →
5. Language model (LLaMA) to generate captions.
- External memory is a pluggable module that can be updated without retraining the main model.

– Front-end UI (Browser)
– Inference API Server (search.py)
– Vision Encoder module (models/blip2.py, eva_vit)
– Q-Former module (models/Qformer.py)
– External Memory Store (ext_data/ext_memory_lvis.pkl + retrieval engine in common/dist_utils.py or search.py)
– Prompt Constructor (prompts/prompt_evcap.txt + common/utils.py)
– Language Model (models/modeling_llama.py)
– Training Pipeline (train_evcap.py, dataset/coco_dataset.py, optims.py)
– Evaluation Pipeline (scripts/eval_evcap_*.sh, eval_evcap.py)
– Common Utilities & Configuration (common/)
– Browser → HTTP POST image → Inference API Server
– Server splits path:
- Vision → Vision Encoder → Q-Former → query embeddings → Memory Store lookup → retrieved names
- Assembled prompt = vision tokens + retrieved names → fed into LLM → text caption
– Caption → API response → Browser display
– Training Pipeline: raw images + annotations → DataLoader → Model (Vision + Q-Former + LLM adapter) → loss backprop (common/optims.py)
– Evaluation Pipeline: loads trained checkpoint → runs inference on benchmark sets → outputs metrics
c. Architectural patterns & design principles:
– Modularity: separate vision, retrieval, prompting, language components
– Pluggable memory: external store decoupled from core model
– Registry/Factory (common/registry.py) for dynamic component instantiation
– Config‐driven (common/config.py + environment.yaml) for reproducibility
d. Key technologies & frameworks:
– Python 3, PyTorch for modeling
– Bulma CSS & JavaScript for front-end
– Memory retrieval (e.g., FAISS or precomputed vectors via common/dist_utils.py)
– LLaMA weights (Hugging Face or local) interfaced via modeling_llama.py

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% UI Layer
    subgraph "UI Layer"
        Browser["Browser UI"]:::frontend
    end

    %% Inference Service
    subgraph "Inference Service"
        SearchServer["Inference API Server\nsearch.py"]:::inference
        VisionEncoderBLIP["BLIP2 Encoder\nmodels/blip2.py"]:::inference
        VisionEncoderEVA["EVA-ViT Encoder\nmodels/eva_vit.py"]:::inference
        QFormer["Q-Former\nmodels/Qformer.py"]:::inference
        RetrievalEngine["Retrieval Engine\ncommon/dist_utils.py"]:::inference
        MemoryStore["External Memory Store\next_data/ext_memory_lvis.pkl"]:::datastore
        PromptTemplate["Prompt Template\nprompts/prompt_evcap.txt"]:::inference
        PromptUtils["Prompt Utilities\ncommon/utils.py"]:::inference
        PromptConstructor["Prompt Constructor"]:::inference
        LLMAdapter["LLaMA Adapter\nmodels/modeling_llama.py"]:::inference
    end

    %% Data Stores
    subgraph "Data Stores"
        MemoryStore
    end

    %% Training Pipeline
    subgraph "Training Pipeline"
        TrainEntry["Training Entry\ntrain_evcap.py"]:::pipeline
        DataLoader["COCO Dataset Loader\ndataset/coco_dataset.py"]:::pipeline
        ModelTrain["EVCap Model\n(Vision+Q-Former+LLM)"]:::pipeline
        Optims["Optimizers Config\ncommon/optims.py"]:::pipeline
    end

    %% Evaluation Pipeline
    subgraph "Evaluation Pipeline"
        EvalDriver["Evaluation Driver\neval_evcap.py"]:::pipeline
        ShellCoco["eval_evcap_coco.sh"]:::pipeline
        ShellFlickr["eval_evcap_flickr30k.sh"]:::pipeline
        ShellNoCaps["eval_evcap_nocaps.sh"]:::pipeline
    end

    %% Common Utilities & Config
    subgraph "Common Utilities & Config"
        Config["Config\ncommon/config.py"]:::common
        Logger["Logger\ncommon/logger.py"]:::common
        Registry["Registry/Factory\ncommon/registry.py"]:::common
        GradCAM["Grad-CAM\ncommon/gradcam.py"]:::common
        DistUtils["Distributed Utils\ncommon/dist_utils.py"]:::common
        Utils["Global Utils\ncommon/utils.py"]:::common
        EnvFile["Environment Spec\nenvironment.yaml"]:::common
    end

    %% Inference Flow
    Browser -->|POST image| SearchServer
    SearchServer --> VisionEncoderBLIP
    SearchServer --> VisionEncoderEVA
    VisionEncoderBLIP --> QFormer
    VisionEncoderEVA --> QFormer
    QFormer -->|query embeddings| RetrievalEngine
    RetrievalEngine -->|lookup| MemoryStore
    MemoryStore -->|retrieved names| PromptConstructor
    PromptTemplate -.-> PromptConstructor
    PromptUtils -.-> PromptConstructor
    QFormer --> PromptConstructor
    PromptConstructor --> LLMAdapter
    LLMAdapter -->|caption| SearchServer
    SearchServer -->|response| Browser

    %% Training Flow
    TrainEntry --> DataLoader
    DataLoader --> ModelTrain
    ModelTrain --> Optims

    %% Evaluation Flow
    EvalDriver --> ShellCoco
    EvalDriver --> ShellFlickr
    EvalDriver --> ShellNoCaps

    %% Click Events
    click Browser "https://github.com/jiaxuan-li/evcap/blob/main/index.html"
    click Browser "https://github.com/jiaxuan-li/evcap/blob/main/static/js/index.js"
    click Browser "https://github.com/jiaxuan-li/evcap/blob/main/static/css/index.css"
    click SearchServer "https://github.com/jiaxuan-li/evcap/blob/main/search.py"
    click VisionEncoderBLIP "https://github.com/jiaxuan-li/evcap/blob/main/models/blip2.py"
    click VisionEncoderEVA "https://github.com/jiaxuan-li/evcap/blob/main/models/eva_vit.py"
    click QFormer "https://github.com/jiaxuan-li/evcap/blob/main/models/Qformer.py"
    click RetrievalEngine "https://github.com/jiaxuan-li/evcap/blob/main/common/dist_utils.py"
    click MemoryStore "https://github.com/jiaxuan-li/evcap/blob/main/ext_data/ext_memory_lvis.pkl"
    click PromptTemplate "https://github.com/jiaxuan-li/evcap/blob/main/prompts/prompt_evcap.txt"
    click PromptUtils "https://github.com/jiaxuan-li/evcap/blob/main/common/utils.py"
    click LLMAdapter "https://github.com/jiaxuan-li/evcap/blob/main/models/modeling_llama.py"
    click TrainEntry "https://github.com/jiaxuan-li/evcap/blob/main/train_evcap.py"
    click DataLoader "https://github.com/jiaxuan-li/evcap/blob/main/dataset/coco_dataset.py"
    click Optims "https://github.com/jiaxuan-li/evcap/blob/main/common/optims.py"
    click EvalDriver "https://github.com/jiaxuan-li/evcap/blob/main/eval_evcap.py"
    click ShellCoco "https://github.com/jiaxuan-li/evcap/blob/main/scripts/eval_evcap_coco.sh"
    click ShellFlickr "https://github.com/jiaxuan-li/evcap/blob/main/scripts/eval_evcap_flickr30k.sh"
    click ShellNoCaps "https://github.com/jiaxuan-li/evcap/blob/main/scripts/eval_evcap_nocaps.sh"
    click Config "https://github.com/jiaxuan-li/evcap/blob/main/common/config.py"
    click Logger "https://github.com/jiaxuan-li/evcap/blob/main/common/logger.py"
    click Registry "https://github.com/jiaxuan-li/evcap/blob/main/common/registry.py"
    click GradCAM "https://github.com/jiaxuan-li/evcap/blob/main/common/gradcam.py"
    click DistUtils "https://github.com/jiaxuan-li/evcap/blob/main/common/dist_utils.py"
    click Utils "https://github.com/jiaxuan-li/evcap/blob/main/common/utils.py"
    click EnvFile "https://github.com/jiaxuan-li/evcap/blob/main/environment.yaml"

    %% Styles
    classDef frontend fill:#cce5ff,stroke:#004085,color:#004085;
    classDef inference fill:#d4edda,stroke:#155724,color:#155724;
    classDef datastore fill:#fff3cd,stroke:#856404,color:#856404;
    classDef pipeline fill:#e2e3e5,stroke:#383d41,color:#383d41;
    classDef common fill:#f8f9fa,stroke:#6c757d,color:#6c757d;
```
