# rajdeep712/multimodal-medical-rag architecture

> Architecture diagram of the GitHub repository rajdeep712/multimodal-medical-rag, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-18). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/rajdeep712/multimodal-medical-rag
- Repository: https://github.com/rajdeep712/multimodal-medical-rag

## Overview

This is a multimodal medical-document RAG application: users authenticate, upload medical records, wait for extraction and indexing, then ask questions and inspect cited source passages. The graph separates the frontend workspace, API/authentication, ingestion, chat/retrieval, and persistence/integrations. Major document, image, citation, and chat capabilities are represented, but the README provides no architectural detail; relationships not shown by the sampled files are omitted or marked unverified.

Main entry points: [`Backend/app/ingestion/pipeline.py`](https://github.com/rajdeep712/multimodal-medical-rag/blob/main/Backend/app/ingestion/pipeline.py) (Ingestion orchestration), [`Backend/app/routers/chats.py`](https://github.com/rajdeep712/multimodal-medical-rag/blob/main/Backend/app/routers/chats.py) (Chat routes), [`Backend/app/routers/auth.py`](https://github.com/rajdeep712/multimodal-medical-rag/blob/main/Backend/app/routers/auth.py) (Authentication routes), [`Backend/app/routers/documents.py`](https://github.com/rajdeep712/multimodal-medical-rag/blob/main/Backend/app/routers/documents.py) (Document routes)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_frontend["Frontend workspace"]
  node_authui["Authentication UI<br/>[LoginForm.tsx]"]
  node_authstate["Session state<br/>[useAuthStore.ts]"]
  node_authservice["Auth service<br/>[authService.ts]"]
  node_workspace["Workspace UI<br/>[WorkspacePage.tsx]"]
  node_chatui["Chat and citations<br/>[ChatPanel.tsx]"]
  node_docui["Document viewer"]
  node_docstate["Document state"]
  node_chatstate["Conversation state<br/>[useChatStore.ts]"]
  node_docservice["Document service<br/>[documentService.ts]"]
  node_chatservice["Chat service<br/>[chatService.ts]"]
  node_apiclient["API client<br/>[api.ts]"]
end

subgraph group_api["API and identity"]
  node_authroutes["Authentication routes<br/>[auth.py]"]
  node_documentroutes["Document routes<br/>[documents.py]"]
  node_chatroutes["Chat routes<br/>[chats.py]"]
end

subgraph group_ingest["Document ingestion"]
  node_ingestflow["Ingestion orchestration<br/>[pipeline.py]"]
  node_textingest["Text and table indexing<br/>[text_pipeline.py]"]
  node_imageingest["Image indexing<br/>[image_pipeline.py]"]
  node_contentextract["Content extraction<br/>[content.py]"]
end

subgraph group_chat["Chat and retrieval"]
  node_retrieval["Retrieval<br/>[retrieval.py]"]
  node_llm["Answer generation<br/>[chat_llm.py]"]
end

subgraph group_data["Persistence and integrations"]
  node_db[("Relational database<br/>[database.py]")]
  node_models["Application records<br/>[models.py]"]
  node_qdrant[("Vector index<br/>[qdrant_store.py]")]
  node_auth0["Auth0 client<br/>[auth0.py]"]
  node_aws["AWS document storage<br/>[aws.py]"]
  node_biomed["BioMedCLIP service<br/>[server.py]"]
end

node_user(("Clinician"))

node_user -->|"signs in"| node_authui
node_authui -->|"submits credentials"| node_authstate
node_authstate -->|"calls auth actions"| node_authservice
node_authservice -->|"sends requests"| node_apiclient
node_authservice -->|"calls backend"| node_authroutes
node_authroutes -->|"authenticates users"| node_auth0
node_authroutes -->|"reads and writes"| node_db
node_authroutes -->|"updates user records"| node_models
node_user -.->|"uses workspace"| node_workspace
node_workspace -.->|"presents documents"| node_docui
node_workspace -.->|"presents chat"| node_chatui
node_docui -.->|"uses document state"| node_docstate
node_docstate -->|"loads documents"| node_docservice
node_docservice -->|"sends requests"| node_apiclient
node_chatui -.->|"uses conversation state"| node_chatstate
node_chatstate -->|"calls chat service"| node_chatservice
node_chatservice -->|"sends requests"| node_apiclient
node_user -.->|"uploads records"| node_documentroutes
node_documentroutes -->|"dispatches ingestion"| node_ingestflow
node_ingestflow -->|"extracts content"| node_contentextract
node_ingestflow -->|"indexes text and tables"| node_textingest
node_ingestflow -->|"indexes images"| node_imageingest
node_ingestflow -->|"downloads encrypted package"| node_aws
node_ingestflow -->|"updates ingestion state"| node_db
node_ingestflow -->|"loads document record"| node_models
node_textingest -->|"upserts vectors"| node_qdrant
node_imageingest -->|"upserts image vectors"| node_qdrant
node_user -.->|"asks questions"| node_chatroutes
node_chatroutes -->|"retrieves evidence"| node_retrieval
node_chatroutes -->|"generates answers"| node_llm
node_chatroutes -->|"queries index"| node_qdrant
node_chatroutes -->|"reads and writes chat"| node_db
node_retrieval -.->|"searches vectors"| node_qdrant
node_documentroutes -->|"reads and writes"| node_db
node_documentroutes -->|"uses document records"| node_models
node_textingest -.->|"requests text embeddings"| node_biomed
node_imageingest -.->|"requests image embeddings"| node_biomed

click node_authui "https://github.com/rajdeep712/multimodal-medical-rag/blob/main/Frontend/src/components/auth/LoginForm.tsx"
click node_authstate "https://github.com/rajdeep712/multimodal-medical-rag/blob/main/Frontend/src/stores/useAuthStore.ts"
click node_authservice "https://github.com/rajdeep712/multimodal-medical-rag/blob/main/Frontend/src/services/authService.ts"
click node_workspace "https://github.com/rajdeep712/multimodal-medical-rag/blob/main/Frontend/src/pages/WorkspacePage.tsx"
click node_chatui "https://github.com/rajdeep712/multimodal-medical-rag/blob/main/Frontend/src/components/chat/ChatPanel.tsx"
click node_docui "https://github.com/rajdeep712/multimodal-medical-rag/blob/main/Frontend/src/components/documents/MedicalDocumentRenderer.tsx"
click node_docstate "https://github.com/rajdeep712/multimodal-medical-rag/blob/main/Frontend/src/stores/useDocumentStore.ts"
click node_chatstate "https://github.com/rajdeep712/multimodal-medical-rag/blob/main/Frontend/src/stores/useChatStore.ts"
click node_docservice "https://github.com/rajdeep712/multimodal-medical-rag/blob/main/Frontend/src/services/documentService.ts"
click node_chatservice "https://github.com/rajdeep712/multimodal-medical-rag/blob/main/Frontend/src/services/chatService.ts"
click node_apiclient "https://github.com/rajdeep712/multimodal-medical-rag/blob/main/Frontend/src/services/api.ts"
click node_authroutes "https://github.com/rajdeep712/multimodal-medical-rag/blob/main/Backend/app/routers/auth.py"
click node_documentroutes "https://github.com/rajdeep712/multimodal-medical-rag/blob/main/Backend/app/routers/documents.py"
click node_chatroutes "https://github.com/rajdeep712/multimodal-medical-rag/blob/main/Backend/app/routers/chats.py"
click node_ingestflow "https://github.com/rajdeep712/multimodal-medical-rag/blob/main/Backend/app/ingestion/pipeline.py"
click node_textingest "https://github.com/rajdeep712/multimodal-medical-rag/blob/main/Backend/app/ingestion/text_pipeline.py"
click node_imageingest "https://github.com/rajdeep712/multimodal-medical-rag/blob/main/Backend/app/ingestion/image_pipeline.py"
click node_contentextract "https://github.com/rajdeep712/multimodal-medical-rag/blob/main/Backend/app/ingestion/content.py"
click node_retrieval "https://github.com/rajdeep712/multimodal-medical-rag/blob/main/Backend/app/retrieval.py"
click node_llm "https://github.com/rajdeep712/multimodal-medical-rag/blob/main/Backend/app/chat_llm.py"
click node_db "https://github.com/rajdeep712/multimodal-medical-rag/blob/main/Backend/app/database.py"
click node_models "https://github.com/rajdeep712/multimodal-medical-rag/blob/main/Backend/app/models.py"
click node_qdrant "https://github.com/rajdeep712/multimodal-medical-rag/blob/main/Backend/app/ingestion/qdrant_store.py"
click node_auth0 "https://github.com/rajdeep712/multimodal-medical-rag/blob/main/Backend/app/auth0.py"
click node_aws "https://github.com/rajdeep712/multimodal-medical-rag/blob/main/Backend/app/aws.py"
click node_biomed "https://github.com/rajdeep712/multimodal-medical-rag/blob/main/BioMedCLIP/server.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_authui,node_authstate,node_authservice,node_workspace,node_chatui,node_docui,node_docstate,node_chatstate,node_docservice,node_chatservice,node_apiclient toneBlue
class node_authroutes,node_documentroutes,node_chatroutes toneAmber
class node_ingestflow,node_textingest,node_imageingest,node_contentextract toneMint
class node_retrieval,node_llm toneRose
class node_db,node_models,node_qdrant,node_auth0,node_aws,node_biomed,node_user toneIndigo
```

## Components

### Frontend workspace

- [Authentication UI](https://github.com/rajdeep712/multimodal-medical-rag/blob/main/Frontend/src/components/auth/LoginForm.tsx): `Frontend/src/components/auth/LoginForm.tsx`
- [Session state](https://github.com/rajdeep712/multimodal-medical-rag/blob/main/Frontend/src/stores/useAuthStore.ts): `Frontend/src/stores/useAuthStore.ts`
- [Auth service](https://github.com/rajdeep712/multimodal-medical-rag/blob/main/Frontend/src/services/authService.ts): `Frontend/src/services/authService.ts`
- [Workspace UI](https://github.com/rajdeep712/multimodal-medical-rag/blob/main/Frontend/src/pages/WorkspacePage.tsx): `Frontend/src/pages/WorkspacePage.tsx`
- [Chat and citations](https://github.com/rajdeep712/multimodal-medical-rag/blob/main/Frontend/src/components/chat/ChatPanel.tsx): `Frontend/src/components/chat/ChatPanel.tsx`
- [Document viewer](https://github.com/rajdeep712/multimodal-medical-rag/blob/main/Frontend/src/components/documents/MedicalDocumentRenderer.tsx): `Frontend/src/components/documents/MedicalDocumentRenderer.tsx`
- [Document state](https://github.com/rajdeep712/multimodal-medical-rag/blob/main/Frontend/src/stores/useDocumentStore.ts): `Frontend/src/stores/useDocumentStore.ts`
- [Conversation state](https://github.com/rajdeep712/multimodal-medical-rag/blob/main/Frontend/src/stores/useChatStore.ts): `Frontend/src/stores/useChatStore.ts`
- [Document service](https://github.com/rajdeep712/multimodal-medical-rag/blob/main/Frontend/src/services/documentService.ts): `Frontend/src/services/documentService.ts`
- [Chat service](https://github.com/rajdeep712/multimodal-medical-rag/blob/main/Frontend/src/services/chatService.ts): `Frontend/src/services/chatService.ts`
- [API client](https://github.com/rajdeep712/multimodal-medical-rag/blob/main/Frontend/src/services/api.ts): `Frontend/src/services/api.ts`

### API and identity

- [Authentication routes](https://github.com/rajdeep712/multimodal-medical-rag/blob/main/Backend/app/routers/auth.py): `Backend/app/routers/auth.py`
- [Document routes](https://github.com/rajdeep712/multimodal-medical-rag/blob/main/Backend/app/routers/documents.py): `Backend/app/routers/documents.py`
- [Chat routes](https://github.com/rajdeep712/multimodal-medical-rag/blob/main/Backend/app/routers/chats.py): `Backend/app/routers/chats.py`

### Document ingestion

- [Ingestion orchestration](https://github.com/rajdeep712/multimodal-medical-rag/blob/main/Backend/app/ingestion/pipeline.py): `Backend/app/ingestion/pipeline.py`
- [Text and table indexing](https://github.com/rajdeep712/multimodal-medical-rag/blob/main/Backend/app/ingestion/text_pipeline.py): `Backend/app/ingestion/text_pipeline.py`
- [Image indexing](https://github.com/rajdeep712/multimodal-medical-rag/blob/main/Backend/app/ingestion/image_pipeline.py): `Backend/app/ingestion/image_pipeline.py`
- [Content extraction](https://github.com/rajdeep712/multimodal-medical-rag/blob/main/Backend/app/ingestion/content.py): `Backend/app/ingestion/content.py`

### Chat and retrieval

- [Retrieval](https://github.com/rajdeep712/multimodal-medical-rag/blob/main/Backend/app/retrieval.py): `Backend/app/retrieval.py`
- [Answer generation](https://github.com/rajdeep712/multimodal-medical-rag/blob/main/Backend/app/chat_llm.py): `Backend/app/chat_llm.py`

### Persistence and integrations

- [Relational database](https://github.com/rajdeep712/multimodal-medical-rag/blob/main/Backend/app/database.py): `Backend/app/database.py`
- [Application records](https://github.com/rajdeep712/multimodal-medical-rag/blob/main/Backend/app/models.py): `Backend/app/models.py`
- [Vector index](https://github.com/rajdeep712/multimodal-medical-rag/blob/main/Backend/app/ingestion/qdrant_store.py): `Backend/app/ingestion/qdrant_store.py`
- [Auth0 client](https://github.com/rajdeep712/multimodal-medical-rag/blob/main/Backend/app/auth0.py): `Backend/app/auth0.py`
- [AWS document storage](https://github.com/rajdeep712/multimodal-medical-rag/blob/main/Backend/app/aws.py): `Backend/app/aws.py`
- [BioMedCLIP service](https://github.com/rajdeep712/multimodal-medical-rag/blob/main/BioMedCLIP/server.py): `BioMedCLIP/server.py`

### Other components

- **Clinician**

## Connections

- Clinician → Authentication UI: signs in
- Authentication UI → Session state: submits credentials (evidence: `Frontend/src/pages/AuthPage.tsx`)
- Session state → Auth service: calls auth actions (evidence: `Frontend/src/stores/useAuthStore.ts`)
- Auth service → API client: sends requests (evidence: `Frontend/src/services/authService.ts`)
- Auth service → Authentication routes: calls backend (evidence: `Frontend/src/services/authService.ts`)
- Authentication routes → Auth0 client: authenticates users (evidence: `Backend/app/routers/auth.py`)
- Authentication routes → Relational database: reads and writes (evidence: `Backend/app/routers/auth.py`)
- Authentication routes → Application records: updates user records (evidence: `Backend/app/routers/auth.py`)
- Clinician → Workspace UI: uses workspace
- Workspace UI → Document viewer: presents documents
- Workspace UI → Chat and citations: presents chat
- Document viewer → Document state: uses document state
- Document state → Document service: loads documents (evidence: `Frontend/src/stores/useDocumentStore.ts`)
- Document service → API client: sends requests (evidence: `Frontend/src/services/documentService.ts`)
- Chat and citations → Conversation state: uses conversation state
- Conversation state → Chat service: calls chat service (evidence: `Frontend/src/stores/useChatStore.ts`)
- Chat service → API client: sends requests (evidence: `Frontend/src/services/chatService.ts`)
- Clinician → Document routes: uploads records
- Document routes → Ingestion orchestration: dispatches ingestion (evidence: `Backend/app/routers/documents.py`)
- Ingestion orchestration → Content extraction: extracts content (evidence: `Backend/app/ingestion/pipeline.py`)
- Ingestion orchestration → Text and table indexing: indexes text and tables (evidence: `Backend/app/ingestion/pipeline.py`)
- Ingestion orchestration → Image indexing: indexes images (evidence: `Backend/app/ingestion/pipeline.py`)
- Ingestion orchestration → AWS document storage: downloads encrypted package (evidence: `Backend/app/ingestion/pipeline.py`)
- Ingestion orchestration → Relational database: updates ingestion state (evidence: `Backend/app/ingestion/pipeline.py`)
- Ingestion orchestration → Application records: loads document record (evidence: `Backend/app/ingestion/pipeline.py`)
- Text and table indexing → Vector index: upserts vectors (evidence: `Backend/app/ingestion/text_pipeline.py`)
- Image indexing → Vector index: upserts image vectors (evidence: `Backend/app/ingestion/image_pipeline.py`)
- Clinician → Chat routes: asks questions
- Chat routes → Retrieval: retrieves evidence (evidence: `Backend/app/routers/chats.py`)
- Chat routes → Answer generation: generates answers (evidence: `Backend/app/routers/chats.py`)
- Chat routes → Vector index: queries index (evidence: `Backend/app/routers/chats.py`)
- Chat routes → Relational database: reads and writes chat (evidence: `Backend/app/routers/chats.py`)
- Retrieval → Vector index: searches vectors
- Document routes → Relational database: reads and writes (evidence: `Backend/app/routers/documents.py`)
- Document routes → Application records: uses document records (evidence: `Backend/app/routers/documents.py`)
- Text and table indexing → BioMedCLIP service: requests text embeddings
- Image indexing → BioMedCLIP service: requests image embeddings
