How cubestar1/triage-flow works
Project type and purpose
• This is a full-stack, multimodal medical triage application (“Triage Flow”) that:
• Provides a Next.js/Tailwind CSS frontend
• Exposes a FastAPI-based backend API
• Integrates an image-classification ML pipeline (ResNet18 on DermNet)
• Runs a RAG (FAISS) knowledge base
• Orchestrates two Gemini ADK agents
• Offers voice & video assistants over WebSocket/external APIs
• Persists data in Supabase (PostgreSQL + Auth)
File structure
• Top-level:
• frontend/ → Next.js app (pages, API routes, supabase clients, hooks, UI components, voice‐assistant)
• backend/ → FastAPI service, RAG module, multi-agent version_3_multi_agent package
• classfication-model/ → standalone image classifier training/inference scripts
• Patterns:
• “version_3_multi_agent” shows a JSON-RPC agent orchestration (server, client, models, agents).
• RAG uses FAISS for vector retrieval.
• frontend/app/api/**/* are proxy endpoints to backend.
• Config & infra:
• frontend/env.example, backend/env.example
• Supabase migration SQL, Next.js config, Docker files (if any), deployment targets (Vercel, Render).
README for additional insights
1. Client Layer
– Web Browser (React/Next.js UI)
– Voice/Video Assistant UI
2. Frontend/API Layer
– Next.js pages & components
– Next.js API routes (auth, triage)
– Supabase client library
3. Backend Services
– FastAPI “triage_endpoint” service
– RAG module (FAISS vector store, retrieval logic)
– Multi-Agent Orchestrator (Gemini ADK Agent 1 & 2 processes)
– Classification-Model service (Tensorflow/ResNet scripts)
4. Datastores
– Supabase PostgreSQL (users, triage_history)
– FAISS vector index (local/in-memory or persisted)
5. External APIs
– Google Gemini Live (WebSocket server for voice)
– Heygen API (video avatars)
– OpenAI/Gemini ADK (NLP agents)
– Resend (email verification)
6. Hosting/Deployment
– Vercel (frontend)
– Render (backend FastAPI, WebSocket)
• Browser⇆Next.js API routes (HTTP/REST)
• Next.js API routes→FastAPI API (HTTP)
• FastAPI→ClassificationModel and RAG modules (function calls or internal HTTP)
• FastAPI→Multi-Agent Orchestrator (local library import or JSON-RPC)
• FastAPI→Supabase (Postgres writes/reads via supabase-py)
• Multi-Agent→OpenAI/Gemini ADK (HTTPS)
• VoiceAssistant UI⇆WebSocket Server (wss://.../ws)→Gemini Live API
• Video Assistant UI→Heygen API
c. Architectural patterns & principles:
• Layered architecture (presentation, API gateway, business logic, data)
• Micro-library pattern for multi-agent orchestration
• Modular ML pipelines (classification, retrieval, agentic reasoning)
• Separation of concerns for identity/auth (Supabase) vs. triage logic
d. Technologies & libraries:
• Frontend: Next.js, Tailwind CSS, shadcn/ui, Supabase JS
• Backend: Python 3.9+, FastAPI, uvicorn, google-adk, tensorflow, faiss-cpu, sentence-transformers, python-dotenv
• DB: Supabase (PostgreSQL + Auth + Storage)
• ML/RAG: Tensorflow/ResNet, FAISS, Gemini API, Google ADK
• Voice: wss (Gemini Live), React hooks (useAudioStream)
• Video: Heygen API
Details
• Major components (classification-model folder, version_3_multi_agent sub-packages).
• Do not omit the voice & video assistants and their WebSocket link.
• Deployment targets (Vercel, Render) in the perimeter.