How soumyaritu/echorxaimediassistant works
1. Top-level shape
EchoRx is a two-part system: a React Native mobile app in app/ and a Python FastAPI backend in backend/. The phone app captures the medicine photo and renders the result UI; the backend performs all AI reasoning and returns structured guidance.
2. Client boundary
The app is an Expo-based React Native client (app/App.js, app/index.js, app/app.json). It is intentionally thin: it handles camera/photo capture, the 3-screen flow described in the README, accessibility, and text-to-speech playback, but does not do the AI inference itself.
3. Backend boundary
The backend is a local FastAPI service (backend/server.py) that exposes the HTTP API used by the app. It acts as the orchestration layer for the AI pipeline and is configured through backend/config.py and environment variables such as GEMINI_API_KEY.
4. Multi-agent AI pipeline
The core architecture is a 3-agent chain under backend/agents/:
- labelsense.py extracts what is visible in the image, such as medicine name, expiry, warnings, and confidence.
- safetyguard.py validates and enriches the result with safety rules, dosage limits, expiry checks, and non-medicine hazard handling.
- voiceassistant.py rewrites the output into short, spoken-friendly language.
root.py coordinates the sequence and combines the final response.
5. Data flow
The main flow is: photo on phone → HTTP request to FastAPI → vision/analysis by LabelSense → safety validation by SafetyGuard → narration text by VoiceAssist → JSON response back to app → spoken output in the app. This is the primary path an architecture diagram should show.
6. External AI dependency
The system depends on Google ADK and Gemini 2.5 Flash for the image/agent reasoning. In architectural terms, Gemini is the external intelligence service behind label reading and safety interpretation, while ADK appears to provide the agent framework and orchestration model.
7. Runtime split
The repo spans two runtimes:
- Node/Expo runtime for the mobile client.
- Python/FastAPI runtime for the backend agents.
The README’s local-network setup implies the app talks to the backend over a configurable LAN API URL in app/config.js or app/App.js.
8. Output contract
The backend likely returns a structured medicine guidance payload rather than free text only, because the app shows distinct UI sections such as medicine name, usage, max daily dose, safety check, and confidence. The voice text is derived from the same response so the visual and spoken outputs stay aligned.
9. Safety and trust layer
A major architectural concern is hallucination control. The README describes explicit rules in LabelSenseAgent for expiry extraction and SafetyGuardAgent for real-time validation against the current date. This makes safety not just a prompt concern, but a dedicated subsystem in the backend.
10. Accessibility layer in the client
The mobile app includes a substantial accessibility presentation layer: screen-reader headings, hidden decorative icons, descriptive labels/hints, and a text-to-speech playback control. This is part of the app architecture, not just styling, because it determines how users consume the backend result.
11. No persistence / no database
There is no database, queue, or user account subsystem in this repo. The app appears to be stateless per scan: each medicine scan is analyzed on demand, returned, and spoken, with no long-term storage shown in the tree.
12. Operational shape
The repository is designed for local development and phone-on-LAN usage rather than a hosted production stack. The backend runs on port 8000, and the app points to the developer machine’s IP address, so the architecture diagram should show a direct client-to-local-backend connection rather than cloud hosting.
13. Key modules to represent
For a diagram, the important boxes are:
- Mobile UI / Expo app
- FastAPI API server
- Root agent orchestrator
- LabelSense vision agent
- SafetyGuard safety agent
- VoiceAssist narration agent
- Gemini/Google ADK external service
These are the real architectural boundaries in this codebase.
14. End-user purpose
The system is intentionally optimized for elderly and visually impaired users, so the architecture is not just “scan and classify”; it is “scan, verify, simplify, and speak.” That purpose explains why the backend is multi-agent and why the frontend includes native accessibility and audio controls.
Main entry points: App.js Expo app · server.py FastAPI server · root.py Root agent · labelsense.py LabelSense
Components 14
Mobile client
Expo/React Native app that captures scans and presents accessible results.
- Expo app · Thin mobile shell for the scan flow and result presentation.
app/App.js - App entry · Launches the mobile application.
app/index.js - Client config · Holds the backend endpoint and app runtime settings.
app/config.js - App manifest · Declares the Expo app metadata and build settings.
app/app.json - Scan flow · Handles photo capture and the three-screen scan-to-result experience.
app/App.js - Accessibility + speech · Screen-reader friendly presentation and text-to-speech playback for results.
app/App.js
AI backend
FastAPI service and agent pipeline that analyzes medicine photos.
- FastAPI server · Backend entrypoint that receives image requests from the mobile app.
backend/server.py - Backend config · Loads API keys and service settings for the backend pipeline.
backend/config.py - Root agent · Coordinates the backend analysis chain and assembles the final payload.
backend/agents/root.py - LabelSense · Extracts medicine identity, visible warnings, expiry clues, and confidence.
backend/agents/labelsense.py - SafetyGuard · Applies dosage, expiry, and non-medicine hazard checks.
backend/agents/safetyguard.py - VoiceAssist · Converts structured guidance into short spoken-friendly wording.
backend/agents/voiceassistant.py
External AI
Google ADK and Gemini provide the model-backed reasoning layer.
- Google ADK · Agent orchestration framework used by the backend.
- Gemini 2.5 Flash · External model used for image understanding and reasoning.
Connections 17
- App entry → Expo app: boots · Starts the Expo client runtime.
- App manifest → Expo app: defines · Provides app metadata and launch configuration.
- Client config → FastAPI server: backend URL · Configures the LAN endpoint for scan requests.
- Expo app → Scan flow: captures photo · User takes a medicine photo and enters the scan flow.
- Scan flow → FastAPI server: POST scan · Sends the image to the backend for analysis.
- FastAPI server → Root agent: dispatches · Hands the request to the agent pipeline.
- Backend config → FastAPI server: configures · Supplies API keys and service settings.
- Root agent → LabelSense: 1. analyze · Extracts visible medicine details from the image.
- LabelSense → Google ADK: runs on · Uses the agent framework for orchestration.
- LabelSense → Gemini 2.5 Flash: vision reasoning · Delegates image interpretation to Gemini.
- Root agent → SafetyGuard: 2. validate · Checks safety rules and enriches the findings.
- SafetyGuard → Google ADK: runs on · Uses the agent framework for orchestration.
- SafetyGuard → Gemini 2.5 Flash: policy reasoning · Uses the model for safety interpretation when needed.
- Root agent → VoiceAssist: 3. narrate · Rewrites the result for spoken output.
- VoiceAssist → Google ADK: runs on · Uses the agent framework for orchestration.
- FastAPI server → Expo app: JSON response · Returns structured guidance to the mobile app.
- Expo app → Accessibility + speech: renders / speaks · Presents the result visually and through text-to-speech.