How liyongqi67/grace works
Project Type and Purpose
• This is an open-source research codebase (“GRACE”) for generative cross-modal retrieval built atop the Open-Flamingo multimodal LLM.
• It provides end-to-end pipelines: data processing (Flickr30k/COCO → WebDataset shards), model fine-tuning (both vanilla and DeepSpeed/FSDP variants), and evaluation/inference (pseudo-query generation and identifier-based retrieval).
High-Level Components
A. Data Sources & Storage
• Raw datasets: JSON annotations and image directories (./data/dataset_*.json, ./data/Flickr30K/, ./data/CoCo/).
• Processed WebDataset shards: under ./data/Openflamingo_format/….
• Model checkpoints: ./checkpoints/....
• Results/metrics: results.json.
B. Data Processing Layer (data_process/ + helper scripts)
• convert_flicker30k_to_wds_* scripts: generate image-to-text, image-to-ID, text-to-ID WebDataset shards.
• structured_id.py & get_trie_dict*: build structured/semantic/numeric identifier mappings and trie constraints for generation.
• Shell wrappers (data.sh) to orchestrate multi-step processing.
C. Model Training
1. Open-Flamingo (vanilla)
• Conda env: open_flamingo/environment.yml.
• Training code: open_flamingo/open_flamingo/train/finetuning.py (and variants).
• Torchrun wrapper: finetuning.sh.
• Uses PyTorch, Transformers (MPT-1B), FSDP optional.
2. Open-Flamingo-DeepSpeed
• Conda env: open_flamingo_deepspeed/environment.yml.
• Training code: open_flamingo_deepspeed/open_flamingo/train/train.py.
• DeepSpeed stage 3/FSFP, gradient checkpointing.
• Logging to WandB (--report_to_wandb).
D. Model Evaluation & Inference
• Shared inference code: open_flamingo/open_flamingo/eval/evaluate.py.
• Scripts: eval.sh, open_flamingo/scripts/run_eval.sh, plus DeepSpeed-specific eval scripts.
• Generates pseudo-queries, constrained generation via trie dictionaries, computes retrieval metrics (COCO, OK-VQA, etc.).
E. Utilities & Configuration
• utils.py, get_trie_dict.py: miscellaneous helpers.
• example.py, test.py: sanity checks.
• Requirements files (requirements.txt, requirements-dev.txt).
• Setup scripts (setup.py, Makefile, .gitignore).
Relationships & Data Flow
• Shards → (torchrun + Open-Flamingo or Open-Flamingo-DeepSpeed) → Checkpoint artifacts.
• Checkpoints → Evaluation script → Retrieval results.
• Results → Metrics computation (COCO/OK-VQA modules) → Output JSON.
Architectural Patterns & Principles
• Modular separation: data prep, training, inference each in its own folder.
• Environment isolation: two Conda envs for inference vs. DeepSpeed training.
• Pipeline reproducibility: shell wrappers drive each stage identifier-by-identifier.
• Extensibility: additional identifier types (numeric, string, semantic, structured, atomic) via uniform scripts.
• Use of WebDataset for scalable data loading and sharding.
Essential Elements
• Clear component labels (e.g., “convert_flicker30k_to_wds_i2t.py”).
• Notes on environment activation steps for each stage.