# homelight/gpt-researcher architecture

> Architecture diagram of the GitHub repository homelight/gpt-researcher, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-01-17). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/homelight/gpt-researcher
- Repository: https://github.com/homelight/gpt-researcher

## Overview

##### Main Components

a) Core System Components:
- Frontend Interface (Web UI)
- Backend Server
- Research Agent System
- Document Processing Pipeline
- Multiple LLM Providers
- Multiple Search Retrievers
- Scraping System
- Memory/Context Management

b) External Integrations:
- Various LLM APIs (OpenAI, Anthropic, Google, etc.)
- Search APIs (Tavily, Google, Bing, etc.)
- Document Storage System

**┌─ User Interface Layer ─────────────────────┐**
- Web Interface (HTML/CSS/JS)              │
- PDF/Document Export                      │
- ┬───────────────────────┘
- ↓
**┌─ API Layer ───────────────────────────────┐**
- FastAPI Server                          │
- WebSocket Manager                       │
- ┬───────────────────────┘
- ↓
**┌─ Agent Orchestration Layer ────────────────┐**
- Research Planner                         │
- Execution Agents                         │
- Multi-Agent System (LangGraph)           │
- ┬───────────────────────┘
- ↓
**┌─ Service Layer ───────────────────────────┐**
- ┌─ LLM Provider ─┐ ┌─ Retrievers ───────┐ │
- OpenAI      │ │ - Tavily           │ │
- Anthropic   │ │ - Google           │ │
- Google      │ │ - Bing             │ │
- Others      │ │ - Others           │ │
- ┘ └──────────────────────┘ │
- ┌─ Scrapers ────┐ ┌─ Document Process ──┐ │
- BeautifulSoup│ │ - PDF              │ │
- Newspaper   │ │ - Text              │ │
- Others      │ │ - HTML              │ │
- ┘ └──────────────────────┘ │
- ┘

##### Key Elements to Highlight

- Blue: Core system components
- Green: External API integrations
- Yellow: Processing components
- Purple: Storage/Memory components

- Rectangles: Major system components
- Hexagons: External services

##### Important Flows

- Research Flow:
1. User Input → Planning Agent → Research Questions
2. Questions → Parallel Execution Agents → Web Scraping
3. Scraped Content → Summarization → Report Generation

- Document Processing Flow:
1. Document Input → Processing Pipeline → Structured Content
2. Content → Context Management → Research Integration

##### Notes

- Version information and date
- Brief descriptions for main components

This system design should effectively communicate the sophisticated architecture of the GPT Researcher while maintaining clarity and providing sufficient detail for implementation understanding.

## Diagram (Mermaid)

```mermaid
graph TB
    %% Styles
    classDef frontend fill:#e1f5fe,stroke:#01579b
    classDef backend fill:#e8f5e9,stroke:#1b5e20
    classDef agent fill:#fff3e0,stroke:#e65100
    classDef service fill:#f3e5f5,stroke:#4a148c
    classDef storage fill:#fce4ec,stroke:#880e4f
    classDef external fill:#efebe9,stroke:#3e2723
    
    %% User Interface Layer
    subgraph UI["User Interface Layer"]
        WebUI["Web Interface"]
        PDFExport["PDF/Document Export"]
    end
    
    %% API Layer
    subgraph API["API Layer"]
        FastAPI["FastAPI Server"]
        WebSocket["WebSocket Manager"]
    end
    
    %% Agent Orchestration Layer
    subgraph Agents["Agent Orchestration Layer"]
        Planner["Research Planner"]
        ExecutionAgents["Execution Agents"]
        MultiAgent["Multi-Agent System"]
    end
    
    %% Service Layer
    subgraph Services["Service Layer"]
        %% LLM Providers
        subgraph LLM["LLM Providers"]
            OpenAI["OpenAI"]
            Anthropic["Anthropic"]
            Google["Google"]
            Mistral["Mistral"]
        end
        
        %% Retrievers
        subgraph Search["Search Retrievers"]
            Tavily["Tavily"]
            GoogleSearch["Google Search"]
            Bing["Bing"]
            SerpAPI["SerpAPI"]
        end
        
        %% Scrapers
        subgraph Scraper["Scraping System"]
            BS["BeautifulSoup"]
            Newspaper["Newspaper"]
            WebLoader["Web Base Loader"]
        end
        
        %% Document Processing
        subgraph DocProcess["Document Processing"]
            PDFProcess["PDF Processing"]
            TextProcess["Text Processing"]
            HTMLProcess["HTML Processing"]
        end
        
        %% Memory Management
        Memory["Memory Management"]
        Context["Context Management"]
    end
    
    %% Connections
    WebUI --> FastAPI
    PDFExport --> FastAPI
    FastAPI --> WebSocket
    WebSocket --> Planner
    Planner --> ExecutionAgents
    ExecutionAgents --> MultiAgent
    MultiAgent --> LLM
    MultiAgent --> Search
    Search --> Scraper
    Scraper --> DocProcess
    DocProcess --> Memory
    Memory --> Context
    Context --> MultiAgent
    
    %% Click Events
    click WebUI "https://github.com/homelight/gpt-researcher/blob/master/frontend/index.html"
    click FastAPI "https://github.com/homelight/gpt-researcher/blob/master/backend/server.py"
    click WebSocket "https://github.com/homelight/gpt-researcher/blob/master/backend/websocket_manager.py"
    click Planner "https://github.com/homelight/gpt-researcher/tree/master/gpt_researcher/master"
    click ExecutionAgents "https://github.com/homelight/gpt-researcher/tree/master/multi_agents/agents"
    click DocProcess "https://github.com/homelight/gpt-researcher/tree/master/scraping/processing"
    click LLM "https://github.com/homelight/gpt-researcher/tree/master/gpt_researcher/llm_provider"
    click OpenAI "https://github.com/homelight/gpt-researcher/tree/master/gpt_researcher/llm_provider/openai"
    click Anthropic "https://github.com/homelight/gpt-researcher/tree/master/gpt_researcher/llm_provider/anthropic"
    click Google "https://github.com/homelight/gpt-researcher/tree/master/gpt_researcher/llm_provider/google"
    click Mistral "https://github.com/homelight/gpt-researcher/tree/master/gpt_researcher/llm_provider/mistral"
    click Search "https://github.com/homelight/gpt-researcher/tree/master/gpt_researcher/retrievers"
    click Tavily "https://github.com/homelight/gpt-researcher/tree/master/gpt_researcher/retrievers/tavily"
    click GoogleSearch "https://github.com/homelight/gpt-researcher/tree/master/gpt_researcher/retrievers/google"
    click Bing "https://github.com/homelight/gpt-researcher/tree/master/gpt_researcher/retrievers/bing"
    click SerpAPI "https://github.com/homelight/gpt-researcher/tree/master/gpt_researcher/retrievers/serpapi"
    click Scraper "https://github.com/homelight/gpt-researcher/tree/master/gpt_researcher/scraper"
    click BS "https://github.com/homelight/gpt-researcher/tree/master/gpt_researcher/scraper/beautiful_soup"
    click Newspaper "https://github.com/homelight/gpt-researcher/tree/master/gpt_researcher/scraper/newspaper"
    click WebLoader "https://github.com/homelight/gpt-researcher/tree/master/gpt_researcher/scraper/web_base_loader"
    click Memory "https://github.com/homelight/gpt-researcher/tree/master/gpt_researcher/memory"
    click Context "https://github.com/homelight/gpt-researcher/tree/master/gpt_researcher/context"
    
    %% Style Applications
    class WebUI,PDFExport frontend
    class FastAPI,WebSocket backend
    class Planner,ExecutionAgents,MultiAgent agent
    class OpenAI,Anthropic,Google,Mistral,Tavily,GoogleSearch,Bing,SerpAPI external
    class BS,Newspaper,WebLoader,PDFProcess,TextProcess,HTMLProcess service
    class Memory,Context storage
```
