How rajdeep712/multimodal-medical-rag works
This is a multimodal medical-document RAG application: users authenticate, upload medical records, wait for extraction and indexing, then ask questions and inspect cited source passages. The graph separates the frontend workspace, API/authentication, ingestion, chat/retrieval, and persistence/integrations. Major document, image, citation, and chat capabilities are represented, but the README provides no architectural detail; relationships not shown by the sampled files are omitted or marked unverified.
Main entry points: pipeline.py Ingestion orchestration · chats.py Chat routes · auth.py Authentication routes · documents.py Document routes
Components 27
Frontend workspace
- Authentication UI
Frontend/src/components/auth/LoginForm.tsx - Session state
Frontend/src/stores/useAuthStore.ts - Auth service
Frontend/src/services/authService.ts - Workspace UI
Frontend/src/pages/WorkspacePage.tsx - Chat and citations
Frontend/src/components/chat/ChatPanel.tsx - Document viewer
Frontend/src/components/documents/MedicalDocumentRenderer.tsx - Document state
Frontend/src/stores/useDocumentStore.ts - Conversation state
Frontend/src/stores/useChatStore.ts - Document service
Frontend/src/services/documentService.ts - Chat service
Frontend/src/services/chatService.ts - API client
Frontend/src/services/api.ts
API and identity
- Authentication routes
Backend/app/routers/auth.py - Document routes
Backend/app/routers/documents.py - Chat routes
Backend/app/routers/chats.py
Document ingestion
- Ingestion orchestration
Backend/app/ingestion/pipeline.py - Text and table indexing
Backend/app/ingestion/text_pipeline.py - Image indexing
Backend/app/ingestion/image_pipeline.py - Content extraction
Backend/app/ingestion/content.py
Chat and retrieval
- Retrieval
Backend/app/retrieval.py - Answer generation
Backend/app/chat_llm.py
Persistence and integrations
- Relational database
Backend/app/database.py - Application records
Backend/app/models.py - Vector index
Backend/app/ingestion/qdrant_store.py - Auth0 client
Backend/app/auth0.py - AWS document storage
Backend/app/aws.py - BioMedCLIP service
BioMedCLIP/server.py
Other components
- Clinician
Connections 37
- Clinician → Authentication UI: signs in
- Authentication UI → Session state: submits credentials
Frontend/src/pages/AuthPage.tsx - Session state → Auth service: calls auth actions
Frontend/src/stores/useAuthStore.ts - Auth service → API client: sends requests
Frontend/src/services/authService.ts - Auth service → Authentication routes: calls backend
Frontend/src/services/authService.ts - Authentication routes → Auth0 client: authenticates users
Backend/app/routers/auth.py - Authentication routes → Relational database: reads and writes
Backend/app/routers/auth.py - Authentication routes → Application records: updates user records
Backend/app/routers/auth.py - Clinician → Workspace UI: uses workspace
- Workspace UI → Document viewer: presents documents
- Workspace UI → Chat and citations: presents chat
- Document viewer → Document state: uses document state
- Document state → Document service: loads documents
Frontend/src/stores/useDocumentStore.ts - Document service → API client: sends requests
Frontend/src/services/documentService.ts - Chat and citations → Conversation state: uses conversation state
- Conversation state → Chat service: calls chat service
Frontend/src/stores/useChatStore.ts - Chat service → API client: sends requests
Frontend/src/services/chatService.ts - Clinician → Document routes: uploads records
- Document routes → Ingestion orchestration: dispatches ingestion
Backend/app/routers/documents.py - Ingestion orchestration → Content extraction: extracts content
Backend/app/ingestion/pipeline.py - Ingestion orchestration → Text and table indexing: indexes text and tables
Backend/app/ingestion/pipeline.py - Ingestion orchestration → Image indexing: indexes images
Backend/app/ingestion/pipeline.py - Ingestion orchestration → AWS document storage: downloads encrypted package
Backend/app/ingestion/pipeline.py - Ingestion orchestration → Relational database: updates ingestion state
Backend/app/ingestion/pipeline.py - Ingestion orchestration → Application records: loads document record
Backend/app/ingestion/pipeline.py - Text and table indexing → Vector index: upserts vectors
Backend/app/ingestion/text_pipeline.py - Image indexing → Vector index: upserts image vectors
Backend/app/ingestion/image_pipeline.py - Clinician → Chat routes: asks questions
- Chat routes → Retrieval: retrieves evidence
Backend/app/routers/chats.py - Chat routes → Answer generation: generates answers
Backend/app/routers/chats.py - Chat routes → Vector index: queries index
Backend/app/routers/chats.py - Chat routes → Relational database: reads and writes chat
Backend/app/routers/chats.py - Retrieval → Vector index: searches vectors
- Document routes → Relational database: reads and writes
Backend/app/routers/documents.py - Document routes → Application records: uses document records
Backend/app/routers/documents.py - Text and table indexing → BioMedCLIP service: requests text embeddings
- Image indexing → BioMedCLIP service: requests image embeddings