How vyshnavnp/gemini-rag-chatbot works
1. Overall shape
- OncoBot is a single-user-request, single-agent oncology assistant. The runtime path is: Streamlit UI → agent orchestrator → tools/cache/memory → response back to UI.
- It is not a multi-agent system; the core intelligence is one LangGraph agent that chooses among tools.
2. User-facing boundary
- app.py is the entry point for the interactive UI.
- It handles chat interactions and uploaded inputs, then hands the request to the agent layer.
- For image/CSV uploads, the UI is only the ingress point; the data is not routed as large tool arguments through the LLM.
3. Agent orchestration layer
- agent/onco_agent.py contains the central LangGraph flow.
- The agent uses Gemini 3.1 Flash Lite Preview as the reasoning model and runs a ReAct-style loop with up to 5 tool iterations per turn.
- The agent is the policy boundary: it decides whether to answer from knowledge retrieval, external search, vision, or ML classification.
4. Semantic cache first
- agent/cache.py implements a ChromaDB-backed response cache.
- Every query is checked before the LLM is called; if cosine similarity is high enough, the system returns immediately and avoids API usage.
- This cache is a major architectural boundary because it can short-circuit the entire agent/tool pipeline.
5. Conversational memory
- agent/memory.py provides session-thread memory via LangGraph checkpointing.
- This lets follow-up questions work without re-supplying prior context.
- Memory is separate from the semantic cache: cache is for repeat/paraphrased queries, memory is for in-session continuity.
6. Tool subsystem: internal knowledge and retrieval
- tools/onco_tools.py holds the main oncology tools.
- The key retrieval path is RAG over the local knowledge base stored in knowledge_base/, which contains MedQuAD-style XML and some PDF documents.
- The embeddings are multilingual, so non-English queries can still match English source material.
7. Tool subsystem: live external research
- tools/external_tools.py exposes live research tools against external services:
- PubMed / NCBI E-utilities
- ClinicalTrials.gov v2
- arXiv
- These tools make the assistant current, but they are separate from the local knowledge-base RAG path.
8. Tool subsystem: vision and medical image analysis
- The app supports multimodal input for images.
- Uploaded image bytes are stored in session-level shared state inside the tool layer, while the image is also passed to Gemini as multimodal content.
- This allows the agent to either reason over the image with Gemini Vision or route to a specialized classifier.
9. Tool subsystem: ML classifiers
- The repo ships local models under models/ for three classification tasks:
- Breast ultrasound classification
- Skin lesion classification
- Cancer type from gene expression
- These models are loaded lazily and run locally on CPU with PyTorch; they are not called through external services.
- The gene-expression classifier also depends on preprocessing artifacts: a scaler and label encoder.
10. Data and state flow for uploads
- app.py reads uploaded files, agent/onco_agent.py stores them in session state, and tools read them directly from shared memory.
- This avoids pushing raw binaries through the LLM tool-call interface.
- After the agent finishes, session upload state is cleared so the next turn starts cleanly.
11. Knowledge-base maintenance
- updater.py maintains the retrieval corpus by re-indexing knowledge_base/.
- The readme indicates incremental updates use SHA-256 hashing and periodic refresh via APScheduler, so the retrieval store stays in sync without duplicating unchanged documents.
- This is an offline/maintenance path separate from normal user queries.
12. Evaluation path
- evaluation/ragas_eval.py is not part of serving traffic.
- It measures answer quality with an LLM-as-judge approach, mainly for faithfulness and relevance.
- Architecturally, this is a sidecar QA loop around the retrieval/agent system, not a runtime dependency.
13. Persistence and packaging
- ChromaDB provides the vector/cached state; model weights and lookup artifacts live locally in the repo.
- The application is containerized with Docker and optionally run via Docker Compose.
- This makes the repo self-contained for local execution and EC2 deployment.
14. Deployment and operational boundary
- CI/CD is defined in .github/workflows/deploy.yml.
- The deployment path is: GitHub push → build Docker image → security scan → push to ECR → deploy to AWS EC2.
- The runtime is a Streamlit app in a Linux container, with security hardening such as non-root execution and restricted capabilities.
15. How to draw the diagram
- The main boxes are: UI, semantic cache, agent orchestrator, tool layer, local knowledge base/vector store, external research APIs, local ML models, and memory.
- The most important arrows are:
- UI → cache → agent
- agent ↔ tools
- tools ↔ ChromaDB/knowledge base
- tools ↔ external APIs
- tools ↔ local PyTorch models
- agent ↔ memory
- uploads → session state → image/CSV tools
Main entry points: onco_tools.py Tool Hub · app.py Streamlit UI · onco_agent.py Onco Agent · cache.py Semantic Cache
Components 17
UI Runtime
Interactive request ingress and serving path
- Streamlit UI · Interactive chat and upload ingress
app.py
Agent Layer
Orchestration, cache, memory, and tool routing
- Semantic Cache · Short-circuits repeat and paraphrased queries before LLM use
agent/cache.py - Onco Agent · Single-agent ReAct loop that selects between retrieval, search, vision, and classifiers
agent/onco_agent.py - Session Memory · LangGraph checkpointed conversational state for follow-ups
agent/memory.py
Data and Tools
Retrieval corpus, live research, multimodal inputs, and local models
- Tool Hub · Main oncology tool surface invoked by the agent
tools/onco_tools.py - Research APIs · Live PubMed, ClinicalTrials.gov, and arXiv lookups
tools/external_tools.py - KB RAG · Local knowledge-base retrieval over MedQuAD-style XML and PDFs
tools/onco_tools.py - Upload State · Holds uploaded image and CSV bytes across the turn
tools/onco_tools.py - Vision Path · Gemini-assisted image reasoning over uploaded medical images
agent/onco_agent.py - Breast Model · Local breast ultrasound classifier loaded on demand
models/oncoscan_bc.pth - Skin Model · Local skin lesion classifier loaded on demand
models/oncoscan_skin.pth - Gene Model · Local cancer-type classifier from gene expression
models/oncotype_bc.pth - Preprocess Artifacts · Scaler and label encoder used by the gene-expression model
models/scaler.pkl - Knowledge Base · XML and PDF source corpus used for retrieval and reindexing
knowledge_base
Ops and Quality
Maintenance, evaluation, and deployment
- Corpus Updater · Re-indexes the corpus with incremental hashing and periodic refresh
updater.py - RAG Eval · LLM-as-judge evaluation for faithfulness and relevance
evaluation/ragas_eval.py - Container Deploy · Docker and GitHub Actions path for container build, scan, and EC2 deploy
Dockerfile
Connections 21
- Streamlit UI → Semantic Cache: check query · Incoming request is checked against semantic cache first
- Semantic Cache → Onco Agent: miss to agent · Uncached queries proceed to the orchestrator
- Semantic Cache → Streamlit UI: hit return · High-similarity matches return directly to the UI
- Streamlit UI → Onco Agent: submit turn · User text and uploads enter the agent boundary
- Onco Agent → Session Memory: read/write · Conversation state is loaded and checkpointed per turn
- Onco Agent → Tool Hub: select tools · ReAct loop invokes tool calls through the main tool hub
- Tool Hub → KB RAG: retrieve · Local retrieval over the knowledge-base corpus
- KB RAG → Knowledge Base: read corpus · Retrieval operates on the local XML and PDF source set
- Tool Hub → Research APIs: live search · External research is routed to live biomedical and literature APIs
- Tool Hub → Upload State: read uploads · Tools consume shared session bytes instead of raw LLM arguments
- Streamlit UI → Upload State: store files · Uploaded images and CSVs are staged in session state
- Tool Hub → Vision Path: analyze image · Image reasoning uses multimodal model support when needed
- Vision Path → Upload State: use image bytes · Vision path consumes stored uploaded bytes
- Tool Hub → Breast Model: classify · Breast ultrasound inference runs locally on CPU
- Tool Hub → Skin Model: classify · Skin lesion inference runs locally on CPU
- Tool Hub → Gene Model: classify · Gene-expression cancer typing runs locally on CPU
- Gene Model → Preprocess Artifacts: preprocess · Model inference depends on scaler and label encoder artifacts
- Corpus Updater → Knowledge Base: reindex · Offline maintenance refreshes the retrieval corpus
- RAG Eval → KB RAG: score outputs · Evaluation wraps the retrieval and answer generation quality
- Container Deploy → Streamlit UI: serve UI · Container build packages the Streamlit serving runtime
- Container Deploy → Onco Agent: package runtime · Deployment ships the agent stack inside the container