How cxcscmu/factmm-rag works
FactMM-RAG is a research pipeline for fact-aware multimodal retrieval-augmented radiology reporting, with a separate non-RAG VQA path. The main workflow parses and labels reports, mines factual report pairs, trains a multimodal retriever, builds nearest-neighbor indexes and LLaVA datasets, then trains, runs, and evaluates report generation. The README documents these stages and optional ANCE/hard-negative and oracle-retrieval branches. Source excerpts confirm key retriever internals; several downstream generator implementations were not sampled, so those links rely on README documentation.
Main entry points: gen_embeddings.py DPR embeddings · DPR/train.py DPR training · gen_topk_pos.py Positive pairs · ANCE/train.py ANCE training
Components 22
Report preparation
- Parse reports
data/parse.py - Clinical labels
data/label.py - Factual scoring
data/factual_mining/build_pos_train/gen_similarity.py - Positive pairs
data/factual_mining/build_pos_train/gen_topk_pos.py
Factual retrieval
- Retriever batches
src/retriever/DPR/data.py - Multimodal encoder
src/retriever/DPR/multi_model.py - DPR training
src/retriever/DPR/train.py - DPR embeddings
src/retriever/DPR/gen_embeddings.py - Hard negatives
src/retriever/DPR/gen_hard_negatives.py - ANCE training
src/retriever/ANCE/train.py - ANCE embeddings
src/retriever/ANCE/gen_embeddings.py
RAG generation
- KNN index
src/generator/knn.py - Oracle KNN
src/generator/knn_ideal.py - RAG dataset builder
src/generator/build_rag_dataset.py - Dataset converter
src/generator/convert_json_or_jsonl.py - LLaVA report generation
src/generator/train_llava.sh - Report inference
src/generator/inference_llava.sh - Report scoring
src/evaluation.py
Non-RAG VQA
- VQA dataset builder
src/generator/build_nonrag_dataset.py - LLaVA VQA
src/generator/vqa/train_llava_vqa.sh
Other components
- Researcher
- Radiology datasets
Connections 24
- Researcher → Radiology datasets: provides
README.md - Radiology datasets → Parse reports: parses
README.md - Parse reports → Clinical labels: annotates
README.md - Clinical labels → Factual scoring: scores reports
README.md - Factual scoring → Positive pairs: ranks pairs
README.md - Positive pairs → Retriever batches: supplies positives
README.md - DPR training → Retriever batches: loads batches
src/retriever/DPR/train.py - DPR training → Multimodal encoder: encodes pairs
src/retriever/DPR/train.py - DPR training → DPR embeddings: saves checkpoint
README.md - DPR embeddings → Multimodal encoder: loads encoder
src/retriever/DPR/gen_embeddings.py - DPR embeddings → Hard negatives: provides embeddings
README.md - Hard negatives → ANCE training: provides negatives
README.md - ANCE training → Retriever batches: uses dataset
src/retriever/ANCE/train.py - ANCE training → ANCE embeddings: saves checkpoint
README.md - DPR embeddings → KNN index: embeds corpus
README.md - KNN index → RAG dataset builder: supplies neighbors
README.md - Positive pairs → Oracle KNN: supplies oracle pairs
README.md - Oracle KNN → RAG dataset builder: supplies oracle index
README.md - RAG dataset builder → Dataset converter: writes dataset
README.md - Dataset converter → LLaVA report generation: prepares training data
README.md - LLaVA report generation → Report inference: trains model
README.md - Report inference → Report scoring: scores outputs
README.md - Radiology datasets → VQA dataset builder: provides reports
README.md - VQA dataset builder → LLaVA VQA: builds VQA data
README.md