How keltoumboukra/twindoc works
Major Components
• Frontend/UI:
• There are two similar directories (“front-end” and “frontend”) that provide the web interface. The Streamlit–based “frontend/app.py” is used to launch an interactive web interface where users (clinicians) can enter queries and view transcripts, thinking processes, and final responses.
• Backend – Orchestration & Agents:
• The “agents” directory contains multiple Python modules such as orchestrator.py, medical_query.py, reza.py, reza_evaluator.py, reza_thinking.py, and reza_transcript.py. These files collectively implement the multi–agent system. The orchestrator acts as the central coordinator that delegates tasks to individual agents (e.g., TranscriptAgent, ThinkingAgent, RezaAgent) that process the incoming medical queries.
• Data Storage & Knowledge Base:
• There are “vectordb” folders (both at the root and inside agents) along with a “knowledge” directory that store vector representations as “.lance” files and related data. This suggests that the system uses a specialized vector database (built with Lancedb/tantivy) to keep track of historical queries, digital twin data, or intermediate reasoning processes.
• External Services & APIs:
• TwinDoc integrates external APIs: the Gemini API (primary) and OpenAI API (fallback). These external services are essential for performing the AI-driven query processing.
• Logging & Monitoring:
• The project integrates Weights & Biases (W&B) extensively. There are dedicated directories (e.g., “agents/wandb” and “wandb” at project root) for logging, which indicates that every query, agent activity, and response is logged for analysis and performance monitoring.
• Error Handling:
• The code includes robust error handling and retry mechanisms (as noted in the README and implied in the file structure via testing modules and scripts such as test_queries).
Relationships and Data Flow
• User Interaction:
• The front-end (Streamlit UI) accepts medical queries from users.
• Query Routing:
• Queries go to the orchestrator (from agents/orchestrator.py), which coordinates between various specialized agents.
• Agent Interactions:
• The Orchestrator delegates to the Transcript Agent, Thinking Agent (that might also include a DuckDuckGo integration or similar lookup), and Reza Agent, which together produce transcript context, reasoning (thinking process), and final responses.
• Data Storage:
• Some agents read/write to the vector databases in “vectordb” or “knowledge” directories, where intermediate representations (like “reza_thinking_process.lance” or “reza_knowledge.lance”) are stored.
• External API Communication:
• During processing, agents route requests to external AI services (Gemini/OpenAI) for generating responses.
• Logging & Monitoring:
• Throughout the process, interactions and errors are recorded to the W&B logs, and the Weave logger (weave_logger.py) might also capture certain logging information.
Architectural Patterns and Design Principles
• Multi–Agent Orchestration:
• The system is based on a multi–agent architecture. Each agent has a dedicated responsibility (i.e., transcript extraction, reasoning, final response formulation) and the orchestrator aggregates the outputs.
• Separation of Concerns:
• Frontend and backend components are clearly separated, as are the different logic modules (e.g., different agents, logging, and knowledge management).
• Robust Error Handling & Resilience:
• The system is designed with automatic retries (for API rate limits and service interruptions) and graceful degradation in mind.
Relevant Technologies
• Programming Language: Python 3.8+
• Frameworks/Libraries:
• Agno for AI agent framework integration.
• Streamlit for the web interface.
• Lancedb and Tantivy for vector database management.
• SQLAlchemy for database interactions (if applicable).
• Weights & Biases (W&B) for logging and monitoring.
• Additional libraries include tenacity (for retries) and python-dotenv (for environment management).