# siddharthyayavaram/caire architecture

> Architecture diagram of the GitHub repository siddharthyayavaram/caire, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-04). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/siddharthyayavaram/caire
- Repository: https://github.com/siddharthyayavaram/caire

## Overview

##### Project Type and Purpose
- CAIRE is a specialized evaluation framework designed to compute cultural relevance scores of images produced by text-to-image models.
- It processes image datasets to perform visual entity linking (extracting and grounding entities via a knowledge base) and then applies a cultural relevance scoring algorithm based on user-specified cultural labels.
- The system leverages pre-trained models, provides retrieval functionalities via several scripts, and outputs matching results which include Wikipedia-based retrieval, lemma matching, and image embeddings.

##### Breakdown of the File Structure (Reflecting Architectural Components)
- Top-level directories indicate a separation between:
- Assets and external documentation (assets, README.md, environment.yaml)
- Core source code (src/) where most modules reside:
∙ The main scripts: main_VEL.py for visual entity linking and main_culture.py for cultural relevance scoring.
∙ The “models” directory contains model loading and configuration logic.
∙ The “scripts” directory holds distinct functionalities: culture_scores.py, disambiguation.py, fetch_wikipedia.py, and retrieval.py.
∙ The “outputs” directory for storing result files (different result pkl files).
- Evaluation resources are stored in the “eval/” folder.
- External model and data requirements are managed via directories such as “checkpoints” and “data”.
- Integration with external repositories is present (e.g., cloned big_vision).

##### Main Components and Their Interactions
a. Data/Input Component:
- Source of image data (provided in src/examples and, more generally, in an external data directory once set up).
- Configuration information with paths set in src/config.py (including DATA_PATH, OUTPUT_PATH, etc.)

b. Preprocessing and Setup:
- A setup script (setup.py) and environment configuration (environment.yaml) that prepare dependencies, download assets (model checkpoints and datasets), and clone the big_vision repository.
- A shell script (src/run_pipeline.sh) that automates the pipeline execution.

c. Core Pipeline/Processing Components:
- Visual Entity Linking (VEL):
- main_VEL.py directs the VEL process.
- Retrieval functionality is handled via scripts (retrieval.py, fetch_wikipedia.py) to access external sources (Wikipedia, BabelNet).
- Cultural Relevance Scoring:
- main_culture.py orchestrates the scoring part.
- Scripts such as culture_scores.py and disambiguation.py are crucial in computing cultural scores based on inputs like lemma embeddings and retrieved text.
- Model Loading:
- modules in src/models (e.g., model_loader.py) that load and manage pre-trained model checkpoints stored in “checkpoints”.

d. Outputs:
- The processed results – which include different match results and rankings – are stored as .pkl files in src/outputs.

##### Architectural Patterns & Design Principles
- The system follows a modular design that separates concerns and logically groups functionalities (e.g., retrieval, processing, scoring).
- The pipeline demonstrates a clear sequential process: setup → visual entity linking → cultural scoring → output generation.
- There is notable use of external configuration (via config.py) to make the system flexible and easily modifiable.
- The integration and use of external dependencies (big_vision, PyTorch, FAISS indices) are key to the system’s design.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data/Ingestion
    subgraph "Data/Input"
        DATA["Image Data (src/examples)"]:::data
    end

    %% Preprocessing & Setup
    subgraph "Preprocessing/Setup"
        SET1["Setup Script (setup.py)"]:::setup
        SET2["Environment Config (environment.yaml)"]:::setup
        SET3["Pipeline Automation (src/run_pipeline.sh)"]:::setup
    end

    %% Configuration & Model Management
    subgraph "Configurations & Models"
        CONFIG["Config (src/config.py)"]:::config
        MODELL["Model Loader (src/models/model_loader.py)"]:::config
    end

    %% Visual Entity Linking (VEL)
    subgraph "Visual Entity Linking"
        VEL["VEL Controller (main_VEL.py)"]:::vel
        RET["Retrieval (src/scripts/retrieval.py)"]:::vel
        WIKI["Fetch Wikipedia (src/scripts/fetch_wikipedia.py)"]:::vel
    end

    %% Cultural Relevance Scoring
    subgraph "Cultural Relevance Scoring"
        CULT["Cultural Scoring (main_culture.py)"]:::cultural
        CSCRIPT["Culture Scores Script (src/scripts/culture_scores.py)"]:::cultural
        DISC["Disambiguation Script (src/scripts/disambiguation.py)"]:::cultural
    end

    %% Output & Visualization
    subgraph "Output & Visualization"
        OUTPUT["Output Storage (src/outputs)"]:::output
        EVAL["Evaluation Notebook (eval/visualization.ipynb)"]:::output
    end

    %% Data Flow Connections
    SET1 --> CONFIG
    SET2 --> CONFIG
    SET3 --> CONFIG

    SET1 --> MODELL
    SET2 --> MODELL
    SET3 --> MODELL

    DATA --> VEL

    CONFIG --> VEL
    CONFIG --> CULT

    MODELL --> VEL
    MODELL --> CULT

    VEL --> RET
    VEL --> WIKI

    VEL --> CULT

    CULT --> CSCRIPT
    CULT --> DISC

    CULT --> OUTPUT
    OUTPUT --> EVAL

    %% Click Events
    click DATA "https://github.com/siddharthyayavaram/caire/tree/main/src/examples"
    click SET1 "https://github.com/siddharthyayavaram/caire/blob/main/setup.py"
    click SET2 "https://github.com/siddharthyayavaram/caire/blob/main/environment.yaml"
    click SET3 "https://github.com/siddharthyayavaram/caire/blob/main/src/run_pipeline.sh"
    click CONFIG "https://github.com/siddharthyayavaram/caire/blob/main/src/config.py"
    click MODELL "https://github.com/siddharthyayavaram/caire/blob/main/src/models/model_loader.py"
    click VEL "https://github.com/siddharthyayavaram/caire/blob/main/src/main_VEL.py"
    click RET "https://github.com/siddharthyayavaram/caire/blob/main/src/scripts/retrieval.py"
    click WIKI "https://github.com/siddharthyayavaram/caire/blob/main/src/scripts/fetch_wikipedia.py"
    click CULT "https://github.com/siddharthyayavaram/caire/blob/main/src/main_culture.py"
    click CSCRIPT "https://github.com/siddharthyayavaram/caire/blob/main/src/scripts/culture_scores.py"
    click DISC "https://github.com/siddharthyayavaram/caire/blob/main/src/scripts/disambiguation.py"
    click OUTPUT "https://github.com/siddharthyayavaram/caire/tree/main/src/outputs"
    click EVAL "https://github.com/siddharthyayavaram/caire/blob/main/eval/visualization.ipynb"

    %% Styles
    classDef data fill:#cce5ff,stroke:#003366,stroke-width:2px;
    classDef setup fill:#fff3cd,stroke:#856404,stroke-width:2px;
    classDef vel fill:#d4edda,stroke:#155724,stroke-width:2px;
    classDef cultural fill:#f8d7da,stroke:#721c24,stroke-width:2px;
    classDef config fill:#e2e3e5,stroke:#6c757d,stroke-width:2px;
    classDef output fill:#d1ecf1,stroke:#0c5460,stroke-width:2px;
```
