# souvikmajumder26/multi-agent-medical-assistant architecture

> Architecture diagram of the GitHub repository souvikmajumder26/multi-agent-medical-assistant, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-02). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/souvikmajumder26/multi-agent-medical-assistant
- Repository: https://github.com/souvikmajumder26/multi-agent-medical-assistant

## Overview

This is a multi-agent medical assistant: users submit text or medical images through the API, and an orchestration graph applies guardrails and routes work to conversational, RAG, web-research, or imaging agents. RAG supports document ingestion, parsing, chunking, query expansion, retrieval, reranking, and grounded response generation; low-confidence answers can hand off to web search. Image results may require human validation, and the API also supports speech input/output. Relationships reflect the supplied excerpts, source index, and README; unsampled implementation details are represented only where documented.

Main entry points: [`agents/rag_agent/__init__.py`](https://github.com/souvikmajumder26/multi-agent-medical-assistant/blob/main/agents/rag_agent/__init__.py) (Medical RAG), [`agents/agent_decision.py`](https://github.com/souvikmajumder26/multi-agent-medical-assistant/blob/main/agents/agent_decision.py) (Agent Router), [`agents/image_analysis_agent/image_classifier.py`](https://github.com/souvikmajumder26/multi-agent-medical-assistant/blob/main/agents/image_analysis_agent/image_classifier.py) (Image Classifier), [`app.py`](https://github.com/souvikmajumder26/multi-agent-medical-assistant/blob/main/app.py) (HTTP API)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_interface["User Interface"]
  node_api["HTTP API<br/>[app.py]"]
  node_speech["Speech Services<br/>[app.py]"]
end

subgraph group_orchestration["Agent Orchestration"]
  node_orchestrator["Agent Router<br/>[agent_decision.py]"]
  node_guardrails["Input Guardrails"]
  node_config["Agent Configuration<br/>[config.py]"]
  node_validation["Human Validation<br/>[app.py]"]
end

subgraph group_knowledge["Medical Knowledge"]
  node_rag["Medical RAG<br/>[__init__.py]"]
  node_parser["Document Parser<br/>[doc_parser.py]"]
  node_content["Content Processor"]
  node_queryexp["Query Expansion<br/>[query_expander.py]"]
  node_vector["Hybrid Retrieval"]
  node_qdrant[("Qdrant Store")]
  node_rerank["Chunk Reranker<br/>[reranker.py]"]
  node_response["RAG Response"]
  node_ingest["Document Ingestion<br/>[ingest_rag_data.py]"]
end

subgraph group_research["Research Search"]
  node_webprocessor["Web Search Processor"]
  node_webagent["Web Search Agent"]
  node_pubmed["PubMed Search<br/>[pubmed_search.py]"]
  node_tavily["Tavily Search<br/>[tavily_search.py]"]
end

subgraph group_imaging["Image Analysis"]
  node_imageclassifier["Image Classifier"]
  node_chest["Chest X-ray Model"]
  node_skin["Skin Lesion Model"]
  node_brain["Brain Tumor Model"]
end

node_user(("User"))
node_elevenlabs["ElevenLabs"]

node_user -->|"submits query"| node_api
node_api -->|"routes chat"| node_orchestrator
node_api -->|"handles audio"| node_speech
node_speech -->|"uses speech API"| node_elevenlabs
node_orchestrator -->|"checks input"| node_guardrails
node_orchestrator -->|"routes queries"| node_rag
node_orchestrator -->|"hands off"| node_webprocessor
node_orchestrator -->|"routes images"| node_imageclassifier
node_orchestrator -->|"requests review"| node_validation
node_rag -->|"parses documents"| node_parser
node_rag -->|"processes content"| node_content
node_rag -->|"expands queries"| node_queryexp
node_rag -->|"retrieves chunks"| node_vector
node_rag -->|"reranks results"| node_rerank
node_rag -->|"generates answer"| node_response
node_vector -->|"reads and writes"| node_qdrant
node_ingest -->|"calls ingestion"| node_rag
node_webprocessor -->|"searches query"| node_webagent
node_webagent -->|"uses search"| node_pubmed
node_webagent -->|"uses search"| node_tavily
node_imageclassifier -->|"dispatches analysis"| node_chest
node_imageclassifier -->|"dispatches analysis"| node_skin
node_imageclassifier -.->|"dispatches analysis"| node_brain
node_config -.->|"configures agents"| node_rag

click node_api "https://github.com/souvikmajumder26/multi-agent-medical-assistant/blob/main/app.py"
click node_speech "https://github.com/souvikmajumder26/multi-agent-medical-assistant/blob/main/app.py"
click node_orchestrator "https://github.com/souvikmajumder26/multi-agent-medical-assistant/blob/main/agents/agent_decision.py"
click node_guardrails "https://github.com/souvikmajumder26/multi-agent-medical-assistant/blob/main/agents/guardrails/local_guardrails.py"
click node_config "https://github.com/souvikmajumder26/multi-agent-medical-assistant/blob/main/config.py"
click node_rag "https://github.com/souvikmajumder26/multi-agent-medical-assistant/blob/main/agents/rag_agent/__init__.py"
click node_parser "https://github.com/souvikmajumder26/multi-agent-medical-assistant/blob/main/agents/rag_agent/doc_parser.py"
click node_content "https://github.com/souvikmajumder26/multi-agent-medical-assistant/blob/main/agents/rag_agent/content_processor.py"
click node_queryexp "https://github.com/souvikmajumder26/multi-agent-medical-assistant/blob/main/agents/rag_agent/query_expander.py"
click node_vector "https://github.com/souvikmajumder26/multi-agent-medical-assistant/blob/main/agents/rag_agent/vectorstore_qdrant.py"
click node_qdrant "https://github.com/souvikmajumder26/multi-agent-medical-assistant/tree/main/data/qdrant_db"
click node_rerank "https://github.com/souvikmajumder26/multi-agent-medical-assistant/blob/main/agents/rag_agent/reranker.py"
click node_response "https://github.com/souvikmajumder26/multi-agent-medical-assistant/blob/main/agents/rag_agent/response_generator.py"
click node_ingest "https://github.com/souvikmajumder26/multi-agent-medical-assistant/blob/main/ingest_rag_data.py"
click node_webprocessor "https://github.com/souvikmajumder26/multi-agent-medical-assistant/blob/main/agents/web_search_processor_agent/web_search_processor.py"
click node_webagent "https://github.com/souvikmajumder26/multi-agent-medical-assistant/blob/main/agents/web_search_processor_agent/web_search_agent.py"
click node_pubmed "https://github.com/souvikmajumder26/multi-agent-medical-assistant/blob/main/agents/web_search_processor_agent/pubmed_search.py"
click node_tavily "https://github.com/souvikmajumder26/multi-agent-medical-assistant/blob/main/agents/web_search_processor_agent/tavily_search.py"
click node_imageclassifier "https://github.com/souvikmajumder26/multi-agent-medical-assistant/blob/main/agents/image_analysis_agent/image_classifier.py"
click node_chest "https://github.com/souvikmajumder26/multi-agent-medical-assistant/blob/main/agents/image_analysis_agent/chest_xray_agent/covid_chest_xray_inference.py"
click node_skin "https://github.com/souvikmajumder26/multi-agent-medical-assistant/blob/main/agents/image_analysis_agent/skin_lesion_agent/skin_lesion_inference.py"
click node_brain "https://github.com/souvikmajumder26/multi-agent-medical-assistant/blob/main/agents/image_analysis_agent/brain_tumor_agent/brain_tumor_inference.py"
click node_validation "https://github.com/souvikmajumder26/multi-agent-medical-assistant/blob/main/app.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_api,node_speech,node_user toneBlue
class node_orchestrator,node_guardrails,node_config,node_validation toneAmber
class node_rag,node_parser,node_content,node_queryexp,node_vector,node_qdrant,node_rerank,node_response,node_ingest toneMint
class node_webprocessor,node_webagent,node_pubmed,node_tavily toneRose
class node_imageclassifier,node_chest,node_skin,node_brain,node_elevenlabs toneIndigo
```

## Components

### User Interface

- [HTTP API](https://github.com/souvikmajumder26/multi-agent-medical-assistant/blob/main/app.py): `app.py`
- [Speech Services](https://github.com/souvikmajumder26/multi-agent-medical-assistant/blob/main/app.py): `app.py`

### Agent Orchestration

- [Agent Router](https://github.com/souvikmajumder26/multi-agent-medical-assistant/blob/main/agents/agent_decision.py): `agents/agent_decision.py`
- [Input Guardrails](https://github.com/souvikmajumder26/multi-agent-medical-assistant/blob/main/agents/guardrails/local_guardrails.py): `agents/guardrails/local_guardrails.py`
- [Agent Configuration](https://github.com/souvikmajumder26/multi-agent-medical-assistant/blob/main/config.py): `config.py`
- [Human Validation](https://github.com/souvikmajumder26/multi-agent-medical-assistant/blob/main/app.py): `app.py`

### Medical Knowledge

- [Medical RAG](https://github.com/souvikmajumder26/multi-agent-medical-assistant/blob/main/agents/rag_agent/__init__.py): `agents/rag_agent/__init__.py`
- [Document Parser](https://github.com/souvikmajumder26/multi-agent-medical-assistant/blob/main/agents/rag_agent/doc_parser.py): `agents/rag_agent/doc_parser.py`
- [Content Processor](https://github.com/souvikmajumder26/multi-agent-medical-assistant/blob/main/agents/rag_agent/content_processor.py): `agents/rag_agent/content_processor.py`
- [Query Expansion](https://github.com/souvikmajumder26/multi-agent-medical-assistant/blob/main/agents/rag_agent/query_expander.py): `agents/rag_agent/query_expander.py`
- [Hybrid Retrieval](https://github.com/souvikmajumder26/multi-agent-medical-assistant/blob/main/agents/rag_agent/vectorstore_qdrant.py): `agents/rag_agent/vectorstore_qdrant.py`
- [Qdrant Store](https://github.com/souvikmajumder26/multi-agent-medical-assistant/tree/main/data/qdrant_db): `data/qdrant_db`
- [Chunk Reranker](https://github.com/souvikmajumder26/multi-agent-medical-assistant/blob/main/agents/rag_agent/reranker.py): `agents/rag_agent/reranker.py`
- [RAG Response](https://github.com/souvikmajumder26/multi-agent-medical-assistant/blob/main/agents/rag_agent/response_generator.py): `agents/rag_agent/response_generator.py`
- [Document Ingestion](https://github.com/souvikmajumder26/multi-agent-medical-assistant/blob/main/ingest_rag_data.py): `ingest_rag_data.py`

### Research Search

- [Web Search Processor](https://github.com/souvikmajumder26/multi-agent-medical-assistant/blob/main/agents/web_search_processor_agent/web_search_processor.py): `agents/web_search_processor_agent/web_search_processor.py`
- [Web Search Agent](https://github.com/souvikmajumder26/multi-agent-medical-assistant/blob/main/agents/web_search_processor_agent/web_search_agent.py): `agents/web_search_processor_agent/web_search_agent.py`
- [PubMed Search](https://github.com/souvikmajumder26/multi-agent-medical-assistant/blob/main/agents/web_search_processor_agent/pubmed_search.py): `agents/web_search_processor_agent/pubmed_search.py`
- [Tavily Search](https://github.com/souvikmajumder26/multi-agent-medical-assistant/blob/main/agents/web_search_processor_agent/tavily_search.py): `agents/web_search_processor_agent/tavily_search.py`

### Image Analysis

- [Image Classifier](https://github.com/souvikmajumder26/multi-agent-medical-assistant/blob/main/agents/image_analysis_agent/image_classifier.py): `agents/image_analysis_agent/image_classifier.py`
- [Chest X-ray Model](https://github.com/souvikmajumder26/multi-agent-medical-assistant/blob/main/agents/image_analysis_agent/chest_xray_agent/covid_chest_xray_inference.py): `agents/image_analysis_agent/chest_xray_agent/covid_chest_xray_inference.py`
- [Skin Lesion Model](https://github.com/souvikmajumder26/multi-agent-medical-assistant/blob/main/agents/image_analysis_agent/skin_lesion_agent/skin_lesion_inference.py): `agents/image_analysis_agent/skin_lesion_agent/skin_lesion_inference.py`
- [Brain Tumor Model](https://github.com/souvikmajumder26/multi-agent-medical-assistant/blob/main/agents/image_analysis_agent/brain_tumor_agent/brain_tumor_inference.py): `agents/image_analysis_agent/brain_tumor_agent/brain_tumor_inference.py`

### Other components

- **User**
- **ElevenLabs**

## Connections

- User → HTTP API: submits query (evidence: `README.md`)
- HTTP API → Agent Router: routes chat (evidence: `app.py`)
- HTTP API → Speech Services: handles audio (evidence: `app.py`)
- Speech Services → ElevenLabs: uses speech API (evidence: `app.py`)
- Agent Router → Input Guardrails: checks input (evidence: `agents/agent_decision.py`)
- Agent Router → Medical RAG: routes queries (evidence: `agents/agent_decision.py`)
- Agent Router → Web Search Processor: hands off (evidence: `agents/agent_decision.py`)
- Agent Router → Image Classifier: routes images (evidence: `agents/agent_decision.py`)
- Agent Router → Human Validation: requests review (evidence: `README.md`)
- Medical RAG → Document Parser: parses documents (evidence: `agents/rag_agent/__init__.py`)
- Medical RAG → Content Processor: processes content (evidence: `agents/rag_agent/__init__.py`)
- Medical RAG → Query Expansion: expands queries (evidence: `agents/rag_agent/__init__.py`)
- Medical RAG → Hybrid Retrieval: retrieves chunks (evidence: `agents/rag_agent/__init__.py`)
- Medical RAG → Chunk Reranker: reranks results (evidence: `agents/rag_agent/__init__.py`)
- Medical RAG → RAG Response: generates answer (evidence: `agents/rag_agent/__init__.py`)
- Hybrid Retrieval → Qdrant Store: reads and writes (evidence: `README.md`)
- Document Ingestion → Medical RAG: calls ingestion (evidence: `ingest_rag_data.py`)
- Web Search Processor → Web Search Agent: searches query (evidence: `agents/web_search_processor_agent/web_search_processor.py`)
- Web Search Agent → PubMed Search: uses search (evidence: `agents/web_search_processor_agent/web_search_agent.py`)
- Web Search Agent → Tavily Search: uses search (evidence: `agents/web_search_processor_agent/web_search_agent.py`)
- Image Classifier → Chest X-ray Model: dispatches analysis (evidence: `agents/image_analysis_agent/__init__.py`)
- Image Classifier → Skin Lesion Model: dispatches analysis (evidence: `agents/image_analysis_agent/__init__.py`)
- Image Classifier → Brain Tumor Model: dispatches analysis
- Agent Configuration → Medical RAG: configures agents
