# vichu-ds/multimodal-graph-rag architecture

> Architecture diagram of the GitHub repository vichu-ds/multimodal-graph-rag, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-07). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/vichu-ds/multimodal-graph-rag
- Repository: https://github.com/vichu-ds/multimodal-graph-rag

## Overview

GraphSight turns research documents into searchable text, figures, and knowledge-graph facts, then answers user questions with cited evidence. The main flow is document ingestion, multimodal indexing, hybrid retrieval, graph expansion, and grounded answer generation. The API owns calls to retrieval, generation, and ingestion; the UI is a separate entry point. README capabilities are included, but the UI-to-API connection is unverified because its source was not excerpted.

Main entry points: [`src/ingest/pipeline.py`](https://github.com/vichu-ds/multimodal-graph-rag/blob/main/src/ingest/pipeline.py) (Ingestion pipeline), [`src/config.py`](https://github.com/vichu-ds/multimodal-graph-rag/blob/main/src/config.py) (Runtime configuration), [`src/api/main.py`](https://github.com/vichu-ds/multimodal-graph-rag/blob/main/src/api/main.py) (FastAPI), [`src/ingest/kg_builder.py`](https://github.com/vichu-ds/multimodal-graph-rag/blob/main/src/ingest/kg_builder.py) (Triplet extraction)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_interface["User interface"]
  node_ui["Streamlit UI<br/>[streamlit_app.py]"]
  node_api["FastAPI<br/>[main.py]"]
end

subgraph group_ingest["Ingestion"]
  node_pipeline["Ingestion pipeline<br/>[pipeline.py]"]
  node_loader["Document loader<br/>[loaders.py]"]
  node_chunker["Text chunker<br/>[chunker.py]"]
  node_embedder["Text and image embeddings<br/>[embeddings.py]"]
  node_kg_builder["Triplet extraction<br/>[kg_builder.py]"]
end

subgraph group_retrieval["Retrieval"]
  node_retriever["Hybrid retriever<br/>[hybrid.py]"]
  node_graph_search["Graph expansion<br/>[graph_search.py]"]
end

subgraph group_generation["Answer generation"]
  node_answer["Cited answer builder<br/>[answer.py]"]
end

subgraph group_infra["Configuration and stores"]
  node_config["Runtime configuration<br/>[config.py]"]
  node_vector_store["Vector store adapter<br/>[vector_store.py]"]
  node_qdrant[("Qdrant")]
  node_neo4j[("Neo4j")]
  node_llm["OpenAI-compatible models"]
end

node_user(("Researcher"))

node_user -->|"asks"| node_ui
node_ui -.->|"submits queries"| node_api
node_api -->|"retrieves evidence"| node_retriever
node_api -->|"generates answer"| node_answer
node_api -->|"starts ingestion"| node_pipeline
node_pipeline -->|"loads files"| node_loader
node_pipeline -->|"chunks text"| node_chunker
node_pipeline -->|"embeds text and images"| node_embedder
node_pipeline -->|"indexes vectors"| node_vector_store
node_pipeline -->|"extracts and indexes facts"| node_kg_builder
node_retriever -->|"embeds query"| node_embedder
node_retriever -->|"searches indexes"| node_vector_store
node_retriever -->|"expands entities"| node_graph_search
node_answer -->|"invokes model"| node_llm
node_kg_builder -->|"extracts triplets"| node_llm
node_kg_builder -->|"writes graph facts"| node_neo4j
node_graph_search -->|"queries graph"| node_neo4j
node_vector_store -->|"reads and writes vectors"| node_qdrant
node_loader -.->|"uses paths"| node_config
node_pipeline -->|"uses ingestion paths"| node_config
node_embedder -->|"reads model settings"| node_config
node_kg_builder -->|"reads model and graph settings"| node_config
node_vector_store -->|"reads store settings"| node_config
node_graph_search -->|"reads graph settings"| node_config
node_answer -->|"reads model settings"| node_config

click node_ui "https://github.com/vichu-ds/multimodal-graph-rag/blob/main/app/streamlit_app.py"
click node_api "https://github.com/vichu-ds/multimodal-graph-rag/blob/main/src/api/main.py"
click node_pipeline "https://github.com/vichu-ds/multimodal-graph-rag/blob/main/src/ingest/pipeline.py"
click node_loader "https://github.com/vichu-ds/multimodal-graph-rag/blob/main/src/ingest/loaders.py"
click node_chunker "https://github.com/vichu-ds/multimodal-graph-rag/blob/main/src/ingest/chunker.py"
click node_embedder "https://github.com/vichu-ds/multimodal-graph-rag/blob/main/src/ingest/embeddings.py"
click node_kg_builder "https://github.com/vichu-ds/multimodal-graph-rag/blob/main/src/ingest/kg_builder.py"
click node_retriever "https://github.com/vichu-ds/multimodal-graph-rag/blob/main/src/retrieval/hybrid.py"
click node_graph_search "https://github.com/vichu-ds/multimodal-graph-rag/blob/main/src/retrieval/graph_search.py"
click node_answer "https://github.com/vichu-ds/multimodal-graph-rag/blob/main/src/generation/answer.py"
click node_config "https://github.com/vichu-ds/multimodal-graph-rag/blob/main/src/config.py"
click node_vector_store "https://github.com/vichu-ds/multimodal-graph-rag/blob/main/src/ingest/vector_store.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_ui,node_api toneBlue
class node_pipeline,node_loader,node_chunker,node_embedder,node_kg_builder toneAmber
class node_retriever,node_graph_search toneMint
class node_answer toneRose
class node_config,node_vector_store,node_qdrant,node_neo4j,node_llm,node_user toneIndigo
```

## Components

### User interface

- [Streamlit UI](https://github.com/vichu-ds/multimodal-graph-rag/blob/main/app/streamlit_app.py): `app/streamlit_app.py`
- [FastAPI](https://github.com/vichu-ds/multimodal-graph-rag/blob/main/src/api/main.py): `src/api/main.py`

### Ingestion

- [Ingestion pipeline](https://github.com/vichu-ds/multimodal-graph-rag/blob/main/src/ingest/pipeline.py): `src/ingest/pipeline.py`
- [Document loader](https://github.com/vichu-ds/multimodal-graph-rag/blob/main/src/ingest/loaders.py): `src/ingest/loaders.py`
- [Text chunker](https://github.com/vichu-ds/multimodal-graph-rag/blob/main/src/ingest/chunker.py): `src/ingest/chunker.py`
- [Text and image embeddings](https://github.com/vichu-ds/multimodal-graph-rag/blob/main/src/ingest/embeddings.py): `src/ingest/embeddings.py`
- [Triplet extraction](https://github.com/vichu-ds/multimodal-graph-rag/blob/main/src/ingest/kg_builder.py): `src/ingest/kg_builder.py`

### Retrieval

- [Hybrid retriever](https://github.com/vichu-ds/multimodal-graph-rag/blob/main/src/retrieval/hybrid.py): `src/retrieval/hybrid.py`
- [Graph expansion](https://github.com/vichu-ds/multimodal-graph-rag/blob/main/src/retrieval/graph_search.py): `src/retrieval/graph_search.py`

### Answer generation

- [Cited answer builder](https://github.com/vichu-ds/multimodal-graph-rag/blob/main/src/generation/answer.py): `src/generation/answer.py`

### Configuration and stores

- [Runtime configuration](https://github.com/vichu-ds/multimodal-graph-rag/blob/main/src/config.py): `src/config.py`
- [Vector store adapter](https://github.com/vichu-ds/multimodal-graph-rag/blob/main/src/ingest/vector_store.py): `src/ingest/vector_store.py`
- **Qdrant**
- **Neo4j**
- **OpenAI-compatible models**

### Other components

- **Researcher**

## Connections

- Researcher → Streamlit UI: asks (evidence: `README.md`)
- Streamlit UI → FastAPI: submits queries
- FastAPI → Hybrid retriever: retrieves evidence (evidence: `src/api/main.py`)
- FastAPI → Cited answer builder: generates answer (evidence: `src/api/main.py`)
- FastAPI → Ingestion pipeline: starts ingestion (evidence: `src/api/main.py`)
- Ingestion pipeline → Document loader: loads files (evidence: `src/ingest/pipeline.py`)
- Ingestion pipeline → Text chunker: chunks text (evidence: `src/ingest/pipeline.py`)
- Ingestion pipeline → Text and image embeddings: embeds text and images (evidence: `src/ingest/pipeline.py`)
- Ingestion pipeline → Vector store adapter: indexes vectors (evidence: `src/ingest/pipeline.py`)
- Ingestion pipeline → Triplet extraction: extracts and indexes facts (evidence: `src/ingest/pipeline.py`)
- Hybrid retriever → Text and image embeddings: embeds query (evidence: `src/retrieval/hybrid.py`)
- Hybrid retriever → Vector store adapter: searches indexes (evidence: `src/retrieval/hybrid.py`)
- Hybrid retriever → Graph expansion: expands entities (evidence: `src/retrieval/hybrid.py`)
- Cited answer builder → OpenAI-compatible models: invokes model (evidence: `src/generation/answer.py`)
- Triplet extraction → OpenAI-compatible models: extracts triplets (evidence: `src/ingest/kg_builder.py`)
- Triplet extraction → Neo4j: writes graph facts (evidence: `src/ingest/kg_builder.py`)
- Graph expansion → Neo4j: queries graph (evidence: `src/retrieval/graph_search.py`)
- Vector store adapter → Qdrant: reads and writes vectors (evidence: `src/ingest/vector_store.py`)
- Document loader → Runtime configuration: uses paths
- Ingestion pipeline → Runtime configuration: uses ingestion paths (evidence: `src/ingest/pipeline.py`)
- Text and image embeddings → Runtime configuration: reads model settings (evidence: `src/ingest/embeddings.py`)
- Triplet extraction → Runtime configuration: reads model and graph settings (evidence: `src/ingest/kg_builder.py`)
- Vector store adapter → Runtime configuration: reads store settings (evidence: `src/ingest/vector_store.py`)
- Graph expansion → Runtime configuration: reads graph settings (evidence: `src/retrieval/graph_search.py`)
- Cited answer builder → Runtime configuration: reads model settings (evidence: `src/generation/answer.py`)
