How homelight/gpt-researcher works
Main Components
a) Core System Components:
• Frontend Interface (Web UI)
• Backend Server
• Research Agent System
• Document Processing Pipeline
• Multiple LLM Providers
• Multiple Search Retrievers
• Scraping System
• Memory/Context Management
b) External Integrations:
• Various LLM APIs (OpenAI, Anthropic, Google, etc.)
• Search APIs (Tavily, Google, Bing, etc.)
• Document Storage System
┌─ User Interface Layer ─────────────────────┐
• Web Interface (HTML/CSS/JS) │
• PDF/Document Export │
• ┬───────────────────────┘
• ↓
┌─ API Layer ───────────────────────────────┐
• FastAPI Server │
• WebSocket Manager │
• ┬───────────────────────┘
• ↓
┌─ Agent Orchestration Layer ────────────────┐
• Research Planner │
• Execution Agents │
• Multi-Agent System (LangGraph) │
• ┬───────────────────────┘
• ↓
┌─ Service Layer ───────────────────────────┐
• ┌─ LLM Provider ─┐ ┌─ Retrievers ───────┐ │
• OpenAI │ │ - Tavily │ │
• Anthropic │ │ - Google │ │
• Google │ │ - Bing │ │
• Others │ │ - Others │ │
• ┘ └──────────────────────┘ │
• ┌─ Scrapers ────┐ ┌─ Document Process ──┐ │
• BeautifulSoup│ │ - PDF │ │
• Newspaper │ │ - Text │ │
• Others │ │ - HTML │ │
• ┘ └──────────────────────┘ │
• ┘
Key Elements to Highlight
• Blue: Core system components
• Green: External API integrations
• Yellow: Processing components
• Purple: Storage/Memory components
• Rectangles: Major system components
• Hexagons: External services
Important Flows
• Research Flow:
1. User Input → Planning Agent → Research Questions
2. Questions → Parallel Execution Agents → Web Scraping
3. Scraped Content → Summarization → Report Generation
• Document Processing Flow:
1. Document Input → Processing Pipeline → Structured Content
2. Content → Context Management → Research Integration
Notes
• Version information and date
• Brief descriptions for main components
This system design should effectively communicate the sophisticated architecture of the GPT Researcher while maintaining clarity and providing sufficient detail for implementation understanding.