How siddharthyayavaram/caire works
Project Type and Purpose
• CAIRE is a specialized evaluation framework designed to compute cultural relevance scores of images produced by text-to-image models.
• It processes image datasets to perform visual entity linking (extracting and grounding entities via a knowledge base) and then applies a cultural relevance scoring algorithm based on user-specified cultural labels.
• The system leverages pre-trained models, provides retrieval functionalities via several scripts, and outputs matching results which include Wikipedia-based retrieval, lemma matching, and image embeddings.
Breakdown of the File Structure (Reflecting Architectural Components)
• Top-level directories indicate a separation between:
• Assets and external documentation (assets, README.md, environment.yaml)
• Core source code (src/) where most modules reside:
∙ The main scripts: main_VEL.py for visual entity linking and main_culture.py for cultural relevance scoring.
∙ The “models” directory contains model loading and configuration logic.
∙ The “scripts” directory holds distinct functionalities: culture_scores.py, disambiguation.py, fetch_wikipedia.py, and retrieval.py.
∙ The “outputs” directory for storing result files (different result pkl files).
• Evaluation resources are stored in the “eval/” folder.
• External model and data requirements are managed via directories such as “checkpoints” and “data”.
• Integration with external repositories is present (e.g., cloned big_vision).
Main Components and Their Interactions
a. Data/Input Component:
• Source of image data (provided in src/examples and, more generally, in an external data directory once set up).
• Configuration information with paths set in src/config.py (including DATA_PATH, OUTPUT_PATH, etc.)
b. Preprocessing and Setup:
• A setup script (setup.py) and environment configuration (environment.yaml) that prepare dependencies, download assets (model checkpoints and datasets), and clone the big_vision repository.
• A shell script (src/run_pipeline.sh) that automates the pipeline execution.
c. Core Pipeline/Processing Components:
• Visual Entity Linking (VEL):
• main_VEL.py directs the VEL process.
• Retrieval functionality is handled via scripts (retrieval.py, fetch_wikipedia.py) to access external sources (Wikipedia, BabelNet).
• Cultural Relevance Scoring:
• main_culture.py orchestrates the scoring part.
• Scripts such as culture_scores.py and disambiguation.py are crucial in computing cultural scores based on inputs like lemma embeddings and retrieved text.
• Model Loading:
• modules in src/models (e.g., model_loader.py) that load and manage pre-trained model checkpoints stored in “checkpoints”.
d. Outputs:
• The processed results – which include different match results and rankings – are stored as .pkl files in src/outputs.
Architectural Patterns & Design Principles
• The system follows a modular design that separates concerns and logically groups functionalities (e.g., retrieval, processing, scoring).
• The pipeline demonstrates a clear sequential process: setup → visual entity linking → cultural scoring → output generation.
• There is notable use of external configuration (via config.py) to make the system flexible and easily modifiable.
• The integration and use of external dependencies (big_vision, PyTorch, FAISS indices) are key to the system’s design.