How vichu-ds/multimodal-graph-rag works
GraphSight turns research documents into searchable text, figures, and knowledge-graph facts, then answers user questions with cited evidence. The main flow is document ingestion, multimodal indexing, hybrid retrieval, graph expansion, and grounded answer generation. The API owns calls to retrieval, generation, and ingestion; the UI is a separate entry point. README capabilities are included, but the UI-to-API connection is unverified because its source was not excerpted.
Main entry points: pipeline.py Ingestion pipeline · config.py Runtime configuration · main.py FastAPI · kg_builder.py Triplet extraction
Components 16
User interface
- Streamlit UI
app/streamlit_app.py - FastAPI
src/api/main.py
Ingestion
- Ingestion pipeline
src/ingest/pipeline.py - Document loader
src/ingest/loaders.py - Text chunker
src/ingest/chunker.py - Text and image embeddings
src/ingest/embeddings.py - Triplet extraction
src/ingest/kg_builder.py
Retrieval
- Hybrid retriever
src/retrieval/hybrid.py - Graph expansion
src/retrieval/graph_search.py
Answer generation
- Cited answer builder
src/generation/answer.py
Configuration and stores
- Runtime configuration
src/config.py - Vector store adapter
src/ingest/vector_store.py - Qdrant
- Neo4j
- OpenAI-compatible models
Other components
- Researcher
Connections 25
- Researcher → Streamlit UI: asks
README.md - Streamlit UI → FastAPI: submits queries
- FastAPI → Hybrid retriever: retrieves evidence
src/api/main.py - FastAPI → Cited answer builder: generates answer
src/api/main.py - FastAPI → Ingestion pipeline: starts ingestion
src/api/main.py - Ingestion pipeline → Document loader: loads files
src/ingest/pipeline.py - Ingestion pipeline → Text chunker: chunks text
src/ingest/pipeline.py - Ingestion pipeline → Text and image embeddings: embeds text and images
src/ingest/pipeline.py - Ingestion pipeline → Vector store adapter: indexes vectors
src/ingest/pipeline.py - Ingestion pipeline → Triplet extraction: extracts and indexes facts
src/ingest/pipeline.py - Hybrid retriever → Text and image embeddings: embeds query
src/retrieval/hybrid.py - Hybrid retriever → Vector store adapter: searches indexes
src/retrieval/hybrid.py - Hybrid retriever → Graph expansion: expands entities
src/retrieval/hybrid.py - Cited answer builder → OpenAI-compatible models: invokes model
src/generation/answer.py - Triplet extraction → OpenAI-compatible models: extracts triplets
src/ingest/kg_builder.py - Triplet extraction → Neo4j: writes graph facts
src/ingest/kg_builder.py - Graph expansion → Neo4j: queries graph
src/retrieval/graph_search.py - Vector store adapter → Qdrant: reads and writes vectors
src/ingest/vector_store.py - Document loader → Runtime configuration: uses paths
- Ingestion pipeline → Runtime configuration: uses ingestion paths
src/ingest/pipeline.py - Text and image embeddings → Runtime configuration: reads model settings
src/ingest/embeddings.py - Triplet extraction → Runtime configuration: reads model and graph settings
src/ingest/kg_builder.py - Vector store adapter → Runtime configuration: reads store settings
src/ingest/vector_store.py - Graph expansion → Runtime configuration: reads graph settings
src/retrieval/graph_search.py - Cited answer builder → Runtime configuration: reads model settings
src/generation/answer.py