# cxcscmu/factmm-rag architecture

> Architecture diagram of the GitHub repository cxcscmu/factmm-rag, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-05). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/cxcscmu/factmm-rag
- Repository: https://github.com/cxcscmu/factmm-rag

## Overview

FactMM-RAG is a research pipeline for fact-aware multimodal retrieval-augmented radiology reporting, with a separate non-RAG VQA path. The main workflow parses and labels reports, mines factual report pairs, trains a multimodal retriever, builds nearest-neighbor indexes and LLaVA datasets, then trains, runs, and evaluates report generation. The README documents these stages and optional ANCE/hard-negative and oracle-retrieval branches. Source excerpts confirm key retriever internals; several downstream generator implementations were not sampled, so those links rely on README documentation.

Main entry points: [`src/retriever/DPR/gen_embeddings.py`](https://github.com/cxcscmu/factmm-rag/blob/main/src/retriever/DPR/gen_embeddings.py) (DPR embeddings), [`src/retriever/DPR/train.py`](https://github.com/cxcscmu/factmm-rag/blob/main/src/retriever/DPR/train.py) (DPR training), [`data/factual_mining/build_pos_train/gen_topk_pos.py`](https://github.com/cxcscmu/factmm-rag/blob/main/data/factual_mining/build_pos_train/gen_topk_pos.py) (Positive pairs), [`src/retriever/ANCE/train.py`](https://github.com/cxcscmu/factmm-rag/blob/main/src/retriever/ANCE/train.py) (ANCE training)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_data["Report preparation"]
  node_parse["Parse reports<br/>[parse.py]"]
  node_label["Clinical labels<br/>[label.py]"]
  node_similarity["Factual scoring<br/>[gen_similarity.py]"]
  node_topk["Positive pairs<br/>[gen_topk_pos.py]"]
end

subgraph group_retrieval["Factual retrieval"]
  node_dpr_data["Retriever batches<br/>[data.py]"]
  node_multimodal["Multimodal encoder<br/>[multi_model.py]"]
  node_dpr_train["DPR training<br/>[train.py]"]
  node_dpr_embeddings["DPR embeddings<br/>[gen_embeddings.py]"]
  node_hard_negatives["Hard negatives"]
  node_ance_train["ANCE training<br/>[train.py]"]
  node_ance_embeddings["ANCE embeddings<br/>[gen_embeddings.py]"]
end

subgraph group_rag["RAG generation"]
  node_knn["KNN index<br/>[knn.py]"]
  node_oracle_knn["Oracle KNN<br/>[knn_ideal.py]"]
  node_rag_dataset["RAG dataset builder"]
  node_convert["Dataset converter"]
  node_llava["LLaVA report generation<br/>[train_llava.sh]"]
  node_inference["Report inference<br/>[inference_llava.sh]"]
  node_evaluation["Report scoring<br/>[evaluation.py]"]
end

subgraph group_vqa["Non-RAG VQA"]
  node_nonrag_builder["VQA dataset builder"]
  node_vqa["LLaVA VQA<br/>[train_llava_vqa.sh]"]
end

node_researcher(("Researcher"))
node_radiology_data[("Radiology datasets")]

node_researcher -->|"provides"| node_radiology_data
node_radiology_data -->|"parses"| node_parse
node_parse -->|"annotates"| node_label
node_label -->|"scores reports"| node_similarity
node_similarity -->|"ranks pairs"| node_topk
node_topk -->|"supplies positives"| node_dpr_data
node_dpr_train -->|"loads batches"| node_dpr_data
node_dpr_train -->|"encodes pairs"| node_multimodal
node_dpr_train -->|"saves checkpoint"| node_dpr_embeddings
node_dpr_embeddings -->|"loads encoder"| node_multimodal
node_dpr_embeddings -->|"provides embeddings"| node_hard_negatives
node_hard_negatives -->|"provides negatives"| node_ance_train
node_ance_train -->|"uses dataset"| node_dpr_data
node_ance_train -->|"saves checkpoint"| node_ance_embeddings
node_dpr_embeddings -->|"embeds corpus"| node_knn
node_knn -->|"supplies neighbors"| node_rag_dataset
node_topk -->|"supplies oracle pairs"| node_oracle_knn
node_oracle_knn -->|"supplies oracle index"| node_rag_dataset
node_rag_dataset -->|"writes dataset"| node_convert
node_convert -->|"prepares training data"| node_llava
node_llava -->|"trains model"| node_inference
node_inference -->|"scores outputs"| node_evaluation
node_radiology_data -->|"provides reports"| node_nonrag_builder
node_nonrag_builder -->|"builds VQA data"| node_vqa

click node_parse "https://github.com/cxcscmu/factmm-rag/blob/main/data/parse.py"
click node_label "https://github.com/cxcscmu/factmm-rag/blob/main/data/label.py"
click node_similarity "https://github.com/cxcscmu/factmm-rag/blob/main/data/factual_mining/build_pos_train/gen_similarity.py"
click node_topk "https://github.com/cxcscmu/factmm-rag/blob/main/data/factual_mining/build_pos_train/gen_topk_pos.py"
click node_dpr_data "https://github.com/cxcscmu/factmm-rag/blob/main/src/retriever/DPR/data.py"
click node_multimodal "https://github.com/cxcscmu/factmm-rag/blob/main/src/retriever/DPR/multi_model.py"
click node_dpr_train "https://github.com/cxcscmu/factmm-rag/blob/main/src/retriever/DPR/train.py"
click node_dpr_embeddings "https://github.com/cxcscmu/factmm-rag/blob/main/src/retriever/DPR/gen_embeddings.py"
click node_hard_negatives "https://github.com/cxcscmu/factmm-rag/blob/main/src/retriever/DPR/gen_hard_negatives.py"
click node_ance_train "https://github.com/cxcscmu/factmm-rag/blob/main/src/retriever/ANCE/train.py"
click node_ance_embeddings "https://github.com/cxcscmu/factmm-rag/blob/main/src/retriever/ANCE/gen_embeddings.py"
click node_knn "https://github.com/cxcscmu/factmm-rag/blob/main/src/generator/knn.py"
click node_oracle_knn "https://github.com/cxcscmu/factmm-rag/blob/main/src/generator/knn_ideal.py"
click node_rag_dataset "https://github.com/cxcscmu/factmm-rag/blob/main/src/generator/build_rag_dataset.py"
click node_convert "https://github.com/cxcscmu/factmm-rag/blob/main/src/generator/convert_json_or_jsonl.py"
click node_llava "https://github.com/cxcscmu/factmm-rag/blob/main/src/generator/train_llava.sh"
click node_inference "https://github.com/cxcscmu/factmm-rag/blob/main/src/generator/inference_llava.sh"
click node_evaluation "https://github.com/cxcscmu/factmm-rag/blob/main/src/evaluation.py"
click node_nonrag_builder "https://github.com/cxcscmu/factmm-rag/blob/main/src/generator/build_nonrag_dataset.py"
click node_vqa "https://github.com/cxcscmu/factmm-rag/blob/main/src/generator/vqa/train_llava_vqa.sh"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_parse,node_label,node_similarity,node_topk toneBlue
class node_dpr_data,node_multimodal,node_dpr_train,node_dpr_embeddings,node_hard_negatives,node_ance_train,node_ance_embeddings,node_radiology_data toneAmber
class node_knn,node_oracle_knn,node_rag_dataset,node_convert,node_llava,node_inference,node_evaluation toneMint
class node_nonrag_builder,node_vqa toneRose
class node_researcher toneIndigo
```

## Components

### Report preparation

- [Parse reports](https://github.com/cxcscmu/factmm-rag/blob/main/data/parse.py): `data/parse.py`
- [Clinical labels](https://github.com/cxcscmu/factmm-rag/blob/main/data/label.py): `data/label.py`
- [Factual scoring](https://github.com/cxcscmu/factmm-rag/blob/main/data/factual_mining/build_pos_train/gen_similarity.py): `data/factual_mining/build_pos_train/gen_similarity.py`
- [Positive pairs](https://github.com/cxcscmu/factmm-rag/blob/main/data/factual_mining/build_pos_train/gen_topk_pos.py): `data/factual_mining/build_pos_train/gen_topk_pos.py`

### Factual retrieval

- [Retriever batches](https://github.com/cxcscmu/factmm-rag/blob/main/src/retriever/DPR/data.py): `src/retriever/DPR/data.py`
- [Multimodal encoder](https://github.com/cxcscmu/factmm-rag/blob/main/src/retriever/DPR/multi_model.py): `src/retriever/DPR/multi_model.py`
- [DPR training](https://github.com/cxcscmu/factmm-rag/blob/main/src/retriever/DPR/train.py): `src/retriever/DPR/train.py`
- [DPR embeddings](https://github.com/cxcscmu/factmm-rag/blob/main/src/retriever/DPR/gen_embeddings.py): `src/retriever/DPR/gen_embeddings.py`
- [Hard negatives](https://github.com/cxcscmu/factmm-rag/blob/main/src/retriever/DPR/gen_hard_negatives.py): `src/retriever/DPR/gen_hard_negatives.py`
- [ANCE training](https://github.com/cxcscmu/factmm-rag/blob/main/src/retriever/ANCE/train.py): `src/retriever/ANCE/train.py`
- [ANCE embeddings](https://github.com/cxcscmu/factmm-rag/blob/main/src/retriever/ANCE/gen_embeddings.py): `src/retriever/ANCE/gen_embeddings.py`

### RAG generation

- [KNN index](https://github.com/cxcscmu/factmm-rag/blob/main/src/generator/knn.py): `src/generator/knn.py`
- [Oracle KNN](https://github.com/cxcscmu/factmm-rag/blob/main/src/generator/knn_ideal.py): `src/generator/knn_ideal.py`
- [RAG dataset builder](https://github.com/cxcscmu/factmm-rag/blob/main/src/generator/build_rag_dataset.py): `src/generator/build_rag_dataset.py`
- [Dataset converter](https://github.com/cxcscmu/factmm-rag/blob/main/src/generator/convert_json_or_jsonl.py): `src/generator/convert_json_or_jsonl.py`
- [LLaVA report generation](https://github.com/cxcscmu/factmm-rag/blob/main/src/generator/train_llava.sh): `src/generator/train_llava.sh`
- [Report inference](https://github.com/cxcscmu/factmm-rag/blob/main/src/generator/inference_llava.sh): `src/generator/inference_llava.sh`
- [Report scoring](https://github.com/cxcscmu/factmm-rag/blob/main/src/evaluation.py): `src/evaluation.py`

### Non-RAG VQA

- [VQA dataset builder](https://github.com/cxcscmu/factmm-rag/blob/main/src/generator/build_nonrag_dataset.py): `src/generator/build_nonrag_dataset.py`
- [LLaVA VQA](https://github.com/cxcscmu/factmm-rag/blob/main/src/generator/vqa/train_llava_vqa.sh): `src/generator/vqa/train_llava_vqa.sh`

### Other components

- **Researcher**
- **Radiology datasets**

## Connections

- Researcher → Radiology datasets: provides (evidence: `README.md`)
- Radiology datasets → Parse reports: parses (evidence: `README.md`)
- Parse reports → Clinical labels: annotates (evidence: `README.md`)
- Clinical labels → Factual scoring: scores reports (evidence: `README.md`)
- Factual scoring → Positive pairs: ranks pairs (evidence: `README.md`)
- Positive pairs → Retriever batches: supplies positives (evidence: `README.md`)
- DPR training → Retriever batches: loads batches (evidence: `src/retriever/DPR/train.py`)
- DPR training → Multimodal encoder: encodes pairs (evidence: `src/retriever/DPR/train.py`)
- DPR training → DPR embeddings: saves checkpoint (evidence: `README.md`)
- DPR embeddings → Multimodal encoder: loads encoder (evidence: `src/retriever/DPR/gen_embeddings.py`)
- DPR embeddings → Hard negatives: provides embeddings (evidence: `README.md`)
- Hard negatives → ANCE training: provides negatives (evidence: `README.md`)
- ANCE training → Retriever batches: uses dataset (evidence: `src/retriever/ANCE/train.py`)
- ANCE training → ANCE embeddings: saves checkpoint (evidence: `README.md`)
- DPR embeddings → KNN index: embeds corpus (evidence: `README.md`)
- KNN index → RAG dataset builder: supplies neighbors (evidence: `README.md`)
- Positive pairs → Oracle KNN: supplies oracle pairs (evidence: `README.md`)
- Oracle KNN → RAG dataset builder: supplies oracle index (evidence: `README.md`)
- RAG dataset builder → Dataset converter: writes dataset (evidence: `README.md`)
- Dataset converter → LLaVA report generation: prepares training data (evidence: `README.md`)
- LLaVA report generation → Report inference: trains model (evidence: `README.md`)
- Report inference → Report scoring: scores outputs (evidence: `README.md`)
- Radiology datasets → VQA dataset builder: provides reports (evidence: `README.md`)
- VQA dataset builder → LLaVA VQA: builds VQA data (evidence: `README.md`)
