# liyongqi67/grace architecture

> Architecture diagram of the GitHub repository liyongqi67/grace, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/liyongqi67/grace
- Repository: https://github.com/liyongqi67/grace

## Overview

##### Project Type and Purpose
- This is an open-source research codebase (“GRACE”) for generative cross-modal retrieval built atop the Open-Flamingo multimodal LLM.
- It provides end-to-end pipelines: data processing (Flickr30k/COCO → WebDataset shards), model fine-tuning (both vanilla and DeepSpeed/FSDP variants), and evaluation/inference (pseudo-query generation and identifier-based retrieval).

##### High-Level Components
A. Data Sources & Storage
- Raw datasets: JSON annotations and image directories (`./data/dataset_*.json`, `./data/Flickr30K/`, `./data/CoCo/`).
- Processed WebDataset shards: under `./data/Openflamingo_format/…`.
- Model checkpoints: `./checkpoints/...`.
- Results/metrics: `results.json`.

B. Data Processing Layer (`data_process/` + helper scripts)
- `convert_flicker30k_to_wds_*` scripts: generate image-to-text, image-to-ID, text-to-ID WebDataset shards.
- `structured_id.py` & `get_trie_dict*`: build structured/semantic/numeric identifier mappings and trie constraints for generation.
- Shell wrappers (`data.sh`) to orchestrate multi-step processing.

C. Model Training
1. Open-Flamingo (vanilla)
- Conda env: `open_flamingo/environment.yml`.
- Training code: `open_flamingo/open_flamingo/train/finetuning.py` (and variants).
- Torchrun wrapper: `finetuning.sh`.
- Uses PyTorch, Transformers (MPT-1B), FSDP optional.

2. Open-Flamingo-DeepSpeed
- Conda env: `open_flamingo_deepspeed/environment.yml`.
- Training code: `open_flamingo_deepspeed/open_flamingo/train/train.py`.
- DeepSpeed stage 3/FSFP, gradient checkpointing.
- Logging to WandB (`--report_to_wandb`).

D. Model Evaluation & Inference
- Shared inference code: `open_flamingo/open_flamingo/eval/evaluate.py`.
- Scripts: `eval.sh`, `open_flamingo/scripts/run_eval.sh`, plus DeepSpeed-specific eval scripts.
- Generates pseudo-queries, constrained generation via trie dictionaries, computes retrieval metrics (COCO, OK-VQA, etc.).

E. Utilities & Configuration
- `utils.py`, `get_trie_dict.py`: miscellaneous helpers.
- `example.py`, `test.py`: sanity checks.
- Requirements files (`requirements.txt`, `requirements-dev.txt`).
- Setup scripts (`setup.py`, Makefile, `.gitignore`).

##### Relationships & Data Flow
- Shards → (torchrun + Open-Flamingo or Open-Flamingo-DeepSpeed) → Checkpoint artifacts.
- Checkpoints → Evaluation script → Retrieval results.
- Results → Metrics computation (COCO/OK-VQA modules) → Output JSON.

##### Architectural Patterns & Principles
- Modular separation: data prep, training, inference each in its own folder.
- Environment isolation: two Conda envs for inference vs. DeepSpeed training.
- Pipeline reproducibility: shell wrappers drive each stage identifier-by-identifier.
- Extensibility: additional identifier types (numeric, string, semantic, structured, atomic) via uniform scripts.
- Use of WebDataset for scalable data loading and sharding.

##### Essential Elements
- Clear component labels (e.g., “convert_flicker30k_to_wds_i2t.py”).
- Notes on environment activation steps for each stage.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data Layer
    RawData["Raw Datasets<br>JSON & Images"]:::data
    RawData -->|feeds into| DataProc

    %% Data Processing Layer
    subgraph "Data Processing" 
        direction TB
        DataProc["Run data.sh"]:::script
        ConvI2T["convert_flicker30k_to_wds_i2t.py"]:::script
        ConvT2ID["convert_flicker30k_to_wds_t2id*.py"]:::script
        StructuredID["structured_id.py"]:::script
        TrieBase["get_trie_dict.py"]:::script
        Trie4ID["get_trie_dict_4id.py"]:::script
        TrieStructured["get_trie_dict_4structured_id.py"]:::script
    end
    DataProc --> ConvI2T
    DataProc --> ConvT2ID
    DataProc --> StructuredID
    DataProc --> TrieBase
    DataProc --> Trie4ID
    DataProc --> TrieStructured
    ConvI2T --> WebShards
    ConvT2ID --> WebShards
    StructuredID --> WebShards
    TrieBase --> WebShards
    Trie4ID --> WebShards
    TrieStructured --> WebShards

    %% Storage: WebDataset Shards
    WebShards["WebDataset Shards"]:::storage

    %% Model Training Layer
    WebShards -->|torchrun + FSDP / WebDataset loader| Training

    subgraph "Model Training" 
        direction TB
        subgraph "Vanilla Open-Flamingo" 
            direction TB
            EnvVanilla["environment.yml"]:::config
            FT1["finetuning.py"]:::code
            FT2["finetuning2.py"]:::code
            FT3["finetuning3.py"]:::code
            RunFT["finetuning.sh"]:::script
            RunTrain["run_train.sh"]:::script
        end
        subgraph "DeepSpeed/FSDP Open-Flamingo" 
            direction TB
            EnvDS["environment.yml"]:::config
            DSTrain["train.py"]:::code
            RunDS1["run_train_deepspeed.sh"]:::script
            RunDS2["run_train_fsdp.sh"]:::script
            RunDSFT["finetuning.sh"]:::script
        end
    end
    Training --> EnvVanilla
    Training --> FT1
    Training --> FT2
    Training --> FT3
    Training --> RunFT
    Training --> RunTrain
    Training --> EnvDS
    Training --> DSTrain
    Training --> RunDS1
    Training --> RunDS2
    Training --> RunDSFT

    %% Storage & Logging
    subgraph "Checkpoints & Logging"
        direction TB
        Checkpoints["Model Checkpoints"]:::storage
        WandB["WandB Logging"]:::storage
    end
    RunTrain --> Checkpoints
    RunFT --> Checkpoints
    FT1 --> Checkpoints
    DSTrain --> Checkpoints
    RunDS1 --> WandB
    RunDS2 --> WandB
    RunDSFT --> Checkpoints

    %% Inference & Evaluation Layer
    Checkpoints -->|load checkpoint| Inference
    subgraph "Evaluation & Inference"
        direction TB
        EvalShared["evaluate.py"]:::code
        RunEval1["run_eval.sh"]:::script
        EvalSh["eval.sh"]:::script
        EvalDS["evaluate.py"]:::code
        RunEvalDS1["run_eval_deepspeed.sh"]:::script
        RunEvalDS2["run_eval_ddp.sh"]:::script
    end
    Inference --> EvalShared
    Inference --> RunEval1
    Inference --> EvalSh
    Inference --> EvalDS
    Inference --> RunEvalDS1
    Inference --> RunEvalDS2

    %% Metrics Utilities
    subgraph "Evaluation Metrics"
        direction TB
        Coco1["coco_metric.py"]:::code
        OKVQA1["ok_vqa_utils.py"]:::code
        Coco2["coco_metric.py"]:::code
        OKVQA2["ok_vqa_utils.py"]:::code
    end
    EvalShared --> Coco1
    EvalShared --> OKVQA1
    EvalDS --> Coco2
    EvalDS --> OKVQA2
    Coco1 --> Results
    OKVQA1 --> Results
    Coco2 --> Results
    OKVQA2 --> Results

    %% Results Storage
    Results["Results & Metrics JSON"]:::storage

    %% Click Events
    click ConvI2T "https://github.com/liyongqi67/grace/blob/main/data_process/convert_flicker30k_to_wds_i2t.py"
    click ConvT2ID "https://github.com/liyongqi67/grace/blob/main/data_process/convert_flicker30k_to_wds_t2id*.py"
    click StructuredID "https://github.com/liyongqi67/grace/blob/main/data_process/structured_id.py"
    click TrieBase "https://github.com/liyongqi67/grace/blob/main/get_trie_dict.py"
    click Trie4ID "https://github.com/liyongqi67/grace/blob/main/get_trie_dict_4id.py"
    click TrieStructured "https://github.com/liyongqi67/grace/blob/main/get_trie_dict_4structured_id.py"
    click DataProc "https://github.com/liyongqi67/grace/blob/main/data.sh"
    click EnvVanilla "https://github.com/liyongqi67/grace/blob/main/open_flamingo/environment.yml"
    click FT1 "https://github.com/liyongqi67/grace/blob/main/open_flamingo/open_flamingo/train/finetuning.py"
    click FT2 "https://github.com/liyongqi67/grace/blob/main/open_flamingo/open_flamingo/train/finetuning2.py"
    click FT3 "https://github.com/liyongqi67/grace/blob/main/open_flamingo/open_flamingo/train/finetuning3.py"
    click RunFT "https://github.com/liyongqi67/grace/blob/main/finetuning.sh"
    click RunTrain "https://github.com/liyongqi67/grace/blob/main/open_flamingo/open_flamingo/scripts/run_train.sh"
    click EnvDS "https://github.com/liyongqi67/grace/blob/main/open_flamingo_deepspeed/environment.yml"
    click DSTrain "https://github.com/liyongqi67/grace/blob/main/open_flamingo_deepspeed/open_flamingo/train/train.py"
    click RunDS1 "https://github.com/liyongqi67/grace/blob/main/open_flamingo_deepspeed/open_flamingo/scripts/run_train_deepspeed.sh"
    click RunDS2 "https://github.com/liyongqi67/grace/blob/main/open_flamingo_deepspeed/open_flamingo/scripts/run_train_fsdp.sh"
    click RunDSFT "https://github.com/liyongqi67/grace/blob/main/open_flamingo_deepspeed/finetuning.sh"
    click EvalShared "https://github.com/liyongqi67/grace/blob/main/open_flamingo/open_flamingo/eval/evaluate.py"
    click RunEval1 "https://github.com/liyongqi67/grace/blob/main/open_flamingo/open_flamingo/scripts/run_eval.sh"
    click EvalSh "https://github.com/liyongqi67/grace/blob/main/eval.sh"
    click EvalDS "https://github.com/liyongqi67/grace/blob/main/open_flamingo_deepspeed/open_flamingo/eval/evaluate.py"
    click RunEvalDS1 "https://github.com/liyongqi67/grace/blob/main/open_flamingo_deepspeed/open_flamingo/scripts/run_eval_deepspeed.sh"
    click RunEvalDS2 "https://github.com/liyongqi67/grace/blob/main/open_flamingo_deepspeed/open_flamingo/scripts/run_eval_ddp.sh"
    click Coco1 "https://github.com/liyongqi67/grace/blob/main/open_flamingo/open_flamingo/eval/coco_metric.py"
    click OKVQA1 "https://github.com/liyongqi67/grace/blob/main/open_flamingo/open_flamingo/eval/ok_vqa_utils.py"
    click Coco2 "https://github.com/liyongqi67/grace/blob/main/open_flamingo_deepspeed/open_flamingo/eval/coco_metric.py"
    click OKVQA2 "https://github.com/liyongqi67/grace/blob/main/open_flamingo_deepspeed/open_flamingo/eval/ok_vqa_utils.py"
    click Checkpoints "https://github.com/liyongqi67/grace/tree/main/./checkpoints/"
    click Results "https://github.com/liyongqi67/grace/blob/main/results.json"

    %% Styles
    classDef data fill:#AED6F1,stroke:#21618C,color:#1B4F72
    classDef script fill:#ABEBC6,stroke:#196F3D,color:#145A32
    classDef storage fill:#D5DBDB,stroke:#7B7D7D,color:#2C3E50
    classDef config fill:#F9E79F,stroke:#B7950B,color:#7D6608
    classDef code fill:#FADBD8,stroke:#C0392B,color:#641E16
    classDef storage fill:#D5DBDB,stroke:#7B7D7D,color:#2C3E50
```
