How aryavaidya08/lens works
HCP Spatial Copilot turns a medication scan into a personalized, patient-aware drug briefing and voice Q&A, with engagement recorded for later familiarity-based answers. The primary workflow runs from the iOS camera through drug identification and the shared API to backend personalization and dossier retrieval; clinicians can also use the React dashboard. Separate branches cover accounts, patient charts, medication reviews, and synced chat/history. Backend startup wiring and selected client calls are evidenced in source excerpts; other product flows and subsystem links are represented from README documentation, not unsampled implementation details.
Main entry points: main.py FastAPI App · drug.py Drug Routes · mongo.py MongoDB · App.jsx Web Dashboard
Components 27
iOS Copilot
- iOS App
Lens/Lens/App/HCPCopilotApp.swift - Camera Detection
Lens/Lens/AR/ARSessionManager.swift - AR HUD
Lens/Lens/AR/HUDOverlayView.swift - Voice Assistant
Lens/Lens/Voice/VoiceAssistantSession.swift - Medication Scan
Lens/Lens/Detection/MedicationDocumentScanner.swift - iOS API Client
Lens/Lens/Networking/APIClient.swift
Backend API
- FastAPI App
backend/app/main.py - Account Routes
backend/app/routes/auth.py - Profile Routes
backend/app/routes/profile.py - Detection Route
backend/app/routes/detect.py - Drug Routes
backend/app/routes/drug.py - Engagement Route
backend/app/routes/engagement.py - Review Routes
backend/app/routes/medication_reviews.py
Drug Intelligence
- Drug Catalog
backend/app/detection/catalog.py - HCP Personalization
backend/app/personalization/scorer.py - Patient Safety Check
backend/app/personalization/patient_check.py - Dossier Ingestion
backend/app/retrieval/ingest.py - Dossier Retrieval
backend/app/retrieval/index.py - Text Embeddings
backend/app/retrieval/embed.py - Answer Generation
backend/app/llm/client.py
Clinical Data
- MongoDB
backend/app/db/mongo.py - Account Storage
backend/app/db/accounts.py - Session Storage
backend/app/db/sessions.py
Web Dashboard
- Web Dashboard
frontend/src/App.jsx - Web API Client
frontend/src/api.js
Other components
- Clinician
- xAI Grok
Connections 35
- Clinician → Camera Detection: scans package
README.md - Camera Detection → iOS API Client: submits barcode/OCR
Lens/Lens/AR/ARSessionManager.swift - iOS API Client → FastAPI App: calls API
Lens/Lens/Networking/APIClient.swift - FastAPI App → Detection Route: registers route
backend/app/main.py - Detection Route → Drug Catalog: resolves identity
- FastAPI App → Drug Routes: registers route
backend/app/main.py - Drug Routes → HCP Personalization: scores familiarity
README.md - Drug Routes → Patient Safety Check: checks patient context
README.md - Drug Routes → Dossier Retrieval: retrieves dossier
README.md - Dossier Retrieval → Text Embeddings: embeds query
backend/app/retrieval/index.py - FastAPI App → Dossier Ingestion: loads dossiers
backend/app/main.py - Dossier Ingestion → Text Embeddings: creates vectors
backend/app/retrieval/ingest.py - Drug Routes → Answer Generation: generates answer
README.md - Answer Generation → xAI Grok: requests completion
README.md - Drug Routes → Engagement Route: logs interaction
README.md - Camera Detection → AR HUD: drives overlay
README.md - Voice Assistant → iOS API Client: sends follow-up
Lens/Lens/Voice/VoiceAssistantSession.swift - Clinician → Voice Assistant: asks by voice
README.md - Clinician → Medication Scan: scans medication list
README.md - Medication Scan → iOS API Client: saves review
Lens/Lens/Views/MedicationReviewsView.swift - FastAPI App → Account Routes: registers route
backend/app/main.py - FastAPI App → Profile Routes: registers route
backend/app/main.py - FastAPI App → Review Routes: registers route
backend/app/main.py - Account Routes → Account Storage: manages accounts
backend/app/routes/auth.py - Account Routes → Session Storage: creates sessions
backend/app/routes/auth.py - Profile Routes → MongoDB: reads and writes charts
README.md - Review Routes → MongoDB: stores reviews
backend/app/routes/medication_reviews.py - Account Storage → MongoDB: persists account data
README.md - Session Storage → MongoDB: persists sessions
backend/app/db/sessions.py - FastAPI App → MongoDB: initializes store
backend/app/main.py - Clinician → Web Dashboard: uses dashboard
README.md - Web Dashboard → Web API Client: requests backend data
frontend/src/App.jsx - Web API Client → FastAPI App: calls shared API
README.md - Web Dashboard → Drug Routes: looks up and asks
frontend/src/App.jsx - Web Dashboard → Profile Routes: manages patients and chats
README.md