# cvlab-kaist/ureca architecture

> Architecture diagram of the GitHub repository cvlab-kaist/ureca, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-07-14). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/cvlab-kaist/ureca
- Repository: https://github.com/cvlab-kaist/ureca

## Overview

##### Project type and purpose
- This is a machine‐learning research codebase (“URECA: Unique Region Caption Anything”) for generating fine‐grained captions on arbitrary image regions.
- It comprises:
- A data curation pipeline (collecting and verifying mask–caption pairs)
- A training framework (model definition, dataset loading, shell wrappers)
- An evaluation suite (multiple domain‐specific benchmarks)
- An interactive demo (Gradio app)

##### File structure
- Top‐level directories:
- data_curation_pipeline/ – Python + shell scripts to build and verify the training dataset
- ureca_model/ – Code to train, evaluate, and package the URECA model
– dataset/, train/, tools/, shell/, ureca/ (model code)
- gradio_demo/ – A small Python app to launch a live demo
- assets/, requirements.txt, README.md

- Configuration & orchestration files:
- run_pipeline.sh, run_data_pipeline.py
- evaluate.sh
- pyproject.toml, requirements.txt

##### README for additional insights
- The pipeline ingests SA‐1B data & human prompts, produces JSONL mask–caption pairs.
- The training module uses PyTorch, custom dataset classes, monkey patches, and shell scripts to launch multi‐stage finetuning.
- The evaluation directory contains per‐benchmark scripts (caption, domain‐specific, VQA, etc.).
- Demo depends on Segment Anything Model (SAM) and serves via Gradio.

a. Main components & their responsibilities:
1) External Data Source
- SA‐1B dataset (images + masks)
- HuggingFace dataset for captions
2) Data Curation Pipeline
- Preprocessing & filtering (utils/preprocess.py)
- Prompt generation (utils/prompts.py)
- GPT‐based verification (gpt_verification.py)
- Output: curated JSONL files
3) Training Framework
- Dataset loaders (train/dataset.py, dataset_packed.py)
- Model code (ureca/model/)
- Training scripts (trainer_dpo.py, mug‐cap_arch_2_train_with_test.py)
- Shell orchestrators (shell/mug‐cap/.../*.sh)
- Output: model checkpoints, weights
4) Evaluation Suite
- Benchmark scripts (ureca_model/eval/.../*.py)
- Aggregate & scoring utilities
5) Demo Application
- Gradio front end (gradio_demo/app.py)
- Loads URECA + SAM checkpoints
6) External Services & Artifacts
- HuggingFace (model & dataset hosting)
- Git LFS (for large files)

b. Relationships & data flow

c. Architectural patterns & design principles
- Pipeline pattern: clear separation between data curation, training, and evaluation
- Modular design: each evaluation benchmark is isolated under its own folder
- Script orchestration: shell scripts for reproducible experiments (multiple stages, architectures)
- Extensibility: tools/ and patch/ directories show plug‐in points for new models or dataset transforms

d. Key technologies & libraries
- Python 3.9, PyTorch 2.5.1, TorchVision, CUDA
- Gradio for demo
- Git LFS for model/data storage
- Shell scripting for experiment management

##### Final recommendations
- Both batch processes (data curation, evaluation) and interactive processes (demo).
- Do not overthink: more granularity makes it easier to assign ownership and understand dependencies.

## Diagram (Mermaid)

```mermaid
flowchart TB
    subgraph "External Data Sources"
        SA1B["SA-1B Dataset"]:::external
        HF["HuggingFace Captions"]:::external
    end

    subgraph "Data Curation Pipeline"
        DataPipeline["Data Curation Pipeline"]:::data
        Preprocess["utils/preprocess.py"]:::data
        Prompts["utils/prompts.py"]:::data
        GPTVerif["gpt_verification.py"]:::data
        JSONL["Mask–Caption JSONL"]:::data
    end

    SA1B --> DataPipeline
    HF --> DataPipeline

    DataPipeline --> Preprocess
    DataPipeline --> Prompts
    DataPipeline --> GPTVerif
    DataPipeline --> JSONL

    subgraph "Training Framework"
        subgraph "Dataset Loaders"
            DatasetLoaders["Dataset Loaders"]:::compute
            DS1["dataset.py"]:::compute
            DS2["dataset_packed.py"]:::compute
        end
        subgraph "Model Definition"
            ModelDef["URECA Model Code"]:::compute
        end
        subgraph "Trainer & Orchestration"
            TrainerDPO["trainer_dpo.py"]:::compute
            Arch2Train["mug-cap_arch_2_train_with_test.py"]:::compute
            Finetune["mug_finetune_with_test.py"]:::compute
            ShellOrch["Shell Scripts"]:::compute
        end
        subgraph "Monkey-Patch Modules"
            Patches["Patch Modules"]:::compute
        end
        subgraph "Training Utilities"
            Tools["Training Tools"]:::compute
        end
        Checkpoints["Model Checkpoints"]:::data
    end

    JSONL --> DatasetLoaders
    DatasetLoaders --> DS1
    DatasetLoaders --> DS2
    DatasetLoaders --> TrainerDPO
    DatasetLoaders --> Arch2Train
    DatasetLoaders --> Finetune
    ModelDef --> TrainerDPO
    ModelDef --> Arch2Train
    ModelDef --> Finetune
    Patches --> TrainerDPO
    Tools --> TrainerDPO
    TrainerDPO --> Checkpoints
    Arch2Train --> Checkpoints
    Finetune --> Checkpoints
    ShellOrch --> Checkpoints

    CheckpointStore["Checkpoint Store (HF / local)"]:::external
    Checkpoints --> CheckpointStore

    subgraph "Evaluation Suite"
        EvalCore["Caption Evaluation"]:::compute
        Domain["Domain-Specific Benchmarks"]:::compute
        Llava["LLaVA Bench"]:::compute
    end

    CheckpointStore --> EvalCore
    CheckpointStore --> Domain
    CheckpointStore --> Llava

    subgraph "Interactive Demo"
        DemoApp["gradio_demo/app.py"]:::demo
        Gallery["Gallery Assets"]:::demo
    end

    SAM["Segment Anything Model (SAM)"]:::external

    CheckpointStore --> DemoApp
    SAM --> DemoApp

    click DataPipeline "https://github.com/cvlab-kaist/ureca/tree/main/data_curation_pipeline/"
    click Preprocess "https://github.com/cvlab-kaist/ureca/blob/main/data_curation_pipeline/utils/preprocess.py"
    click Prompts "https://github.com/cvlab-kaist/ureca/blob/main/data_curation_pipeline/utils/prompts.py"
    click GPTVerif "https://github.com/cvlab-kaist/ureca/blob/main/data_curation_pipeline/gpt_verification.py"
    click DatasetLoaders "https://github.com/cvlab-kaist/ureca/tree/main/ureca_model/ureca/train/"
    click DS1 "https://github.com/cvlab-kaist/ureca/blob/main/ureca_model/ureca/train/dataset.py"
    click DS2 "https://github.com/cvlab-kaist/ureca/blob/main/ureca_model/ureca/train/dataset_packed.py"
    click TrainerDPO "https://github.com/cvlab-kaist/ureca/blob/main/ureca_model/ureca/train/trainer_dpo.py"
    click Arch2Train "https://github.com/cvlab-kaist/ureca/blob/main/ureca_model/ureca/train/mug-cap_arch_2_train_with_test.py"
    click Finetune "https://github.com/cvlab-kaist/ureca/blob/main/ureca_model/ureca/train/mug_finetune_with_test.py"
    click ShellOrch "https://github.com/cvlab-kaist/ureca/tree/main/ureca_model/shell/mug-cap/"
    click ModelDef "https://github.com/cvlab-kaist/ureca/tree/main/ureca_model/ureca/model/"
    click Patches "https://github.com/cvlab-kaist/ureca/tree/main/ureca_model/ureca/patch/"
    click Tools "https://github.com/cvlab-kaist/ureca/tree/main/ureca_model/tools/"
    click EvalCore "https://github.com/cvlab-kaist/ureca/blob/main/ureca_model/eval/caption/evaluate_caption.py"
    click Domain "https://github.com/cvlab-kaist/ureca/tree/main/ureca_model/eval/domain_specific/"
    click Llava "https://github.com/cvlab-kaist/ureca/tree/main/ureca_model/eval/llava_bench/"
    click DemoApp "https://github.com/cvlab-kaist/ureca/blob/main/gradio_demo/app.py"
    click Gallery "https://github.com/cvlab-kaist/ureca/tree/main/gradio_demo/gallery/"

    classDef data fill:#b3cde0,stroke:#6497b1,stroke-width:1px
    classDef compute fill:#ccebc5,stroke:#78c679,stroke-width:1px
    classDef demo fill:#f7e1b5,stroke:#fdae61,stroke-width:1px
    classDef external fill:#e5e5e5,stroke:#999999,stroke-width:1px
```
