# shibing624/deep-research architecture

> Architecture diagram of the GitHub repository shibing624/deep-research, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-06). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/shibing624/deep-research
- Repository: https://github.com/shibing624/deep-research

## Overview

##### Project Type and Purpose
- The project is an AI-powered research assistant written in Python. It performs iterative, deep research on any given topic by combining capabilities of search engine APIs (Serper, MP Search, Tavily) with large language models (LLM). This project supports multiple interfaces (command-line, Gradio web UI, and direct Python module integration) and is intended for both technical users and researchers.
- The README highlights features such as iterative search, automated query generation, multi-language support, and configurable clarification processes. It’s clear that the system is modular and designed to interact with various external services.

##### Analyzing the File Structure
- Top-level files and directories:
- main.py: Entry point for CLI commands, demo, and potentially starting the Gradio interface.
- deep_research_demo.py: A demo script showcasing a complete research flow.
- config.example.yaml: An example configuration file containing key settings (API keys, model hierarchies, search engine options).
- src directory: Contains core modules such as deep_research.py (main logic), config.py (configuration parsing), gradio_chat.py (interface for web UI), and various modules that interface with different search providers (serper_client.py, mp_search_client.py, tavily_client.py). Also includes model_utils, prompts, providers, and search_utils.
- tests directory: Contains unit tests for key components (providers and client integrations).

##### Main Components
a. User Interfaces and Entry Points:
- CLI (via main.py)
- Gradio Web Interface (triggered via main.py demo and handled by src/gradio_chat.py)
- Python Module usage (direct calls to src.deep_research and its asynchronous API)

b. Core Application Components:
- Deep Research Engine (src/deep_research.py): Responsible for managing the research flow, iteratively generating and refining queries, managing context, and calling the LLM.
- Configuration Management (src/config.py): Loads and manages configuration options (API keys, search preferences, context limits, etc.), typically fed from config.yaml.
- Provider Modules (src/providers.py alongside serper_client.py, mp_search_client.py, tavily_client.py): Encapsulate integrations with external search APIs.
- Prompt Generation and Utility Modules (src/prompts.py, src/search_utils.py, src/model_utils.py): Handle the creation of prompts for the LLM and processing of search results.

c. External Services:
- Search Engines: The API endpoints for Serper, MP Search, and Tavily.
- Potential LLM and Web scraping engines (even if only conceptually defined through the deep research flow).

d. Supporting Components:
- Demo and documentation (deep_research_demo.py and docs/ folder)
- Unit Testing Suite (tests/ directory)

##### Relationships and Data Flow
- This query is passed to the Deep Research Engine which:
- Reads configuration settings from the Config module.
- Generates initial prompts using the Prompt module.
- Dispatches queries to the appropriate search provider client based on configuration.
- Receives search results, applies search utility functions to refine or reformat results, and may invoke the LLM for deep context understanding.
- The outputs (final report, iterative research data) are then returned to the user via the chosen interface.
- Additionally, the testing components interact with both the provider clients and core modules to ensure system reliability.

##### Architectural Patterns and Technologies
- The architecture follows a modular design where each concern (search integration, prompt generation, configuration, user interface) is separated into its module under the src directory.
- The use of asynchronous programming (as seen in deep_research_stream) emphasizes non-blocking I/O when interacting with external APIs.
- Integration of a web UI using Gradio demonstrates a separation between backend research logic and frontend presentation.
- Built-in unit tests indicate a test-driven approach for quality assurance.

## Diagram (Mermaid)

```mermaid
graph TD
    subgraph "User Interfaces"
        CLI["CLI (main.py)"]
        Gradio["Gradio Web UI (src/gradio_chat.py)"]
    end

    subgraph "Configuration Manager"
        ConfigModule["Configuration Module (src/config.py)"]
        ConfigFile["Configuration File (config.example.yaml)"]
    end

    subgraph "Prompt & Utility Modules"
        Prompts["Prompt Generation (src/prompts.py)"]
        SearchUtils["Search Utilities (src/search_utils.py)"]
        ModelUtils["Model Utilities (src/model_utils.py)"]
    end

    subgraph "Provider Modules"
        Providers["Provider Integrations (src/providers.py)"]
        Serper["Serper Client (src/serper_client.py)"]
        MP["MP Search Client (src/mp_search_client.py)"]
        Tavily["Tavily Client (src/tavily_client.py)"]
    end

    subgraph "External Services"
        ExtSerper["Serper API"]
        ExtMP["MP Search API"]
        ExtTavily["Tavily API"]
        ExtLLM["LLM Service"]
    end

    subgraph "Demo & Documentation"
        Demo["Demo Script (deep_research_demo.py)"]
        Docs["Project Documentation (docs/project.md)"]
    end

    subgraph "Unit Testing Suite"
        TestsDir["Tests Directory (tests/)"]
        TestProviders["Provider Tests (tests/test_providers.py)"]
        TestSerperTests["Serper Client Tests (tests/test_serper_client.py)"]
    end

    Engine["Deep Research Engine (src/deep_research.py)"]

    %% Data Flow from User Interfaces to Engine
    CLI -->|"Query"| Engine
    Gradio -->|"Query"| Engine

    %% Engine loads configuration
    Engine -->|"loads_config"| ConfigModule
    Engine -->|"loads_config"| ConfigFile

    %% Engine uses prompt and utility modules
    Engine -->|"generates_prompt"| Prompts
    Engine -->|"processes_results"| SearchUtils
    Engine -->|"model_utilization"| ModelUtils

    %% Engine dispatches search request to provider modules
    Engine -->|"dispatch_search"| Providers
    Providers -->|"calls"| Serper
    Providers -->|"calls"| MP
    Providers -->|"calls"| Tavily

    %% Provider modules interact with external services
    Serper -->|"API_request"| ExtSerper
    MP -->|"API_request"| ExtMP
    Tavily -->|"API_request"| ExtTavily

    %% Engine queries LLM externally
    Engine -->|"calls_LLM"| ExtLLM

    %% Final report returned to users
    Engine -->|"final_report"| CLI
    Engine -->|"final_report"| Gradio

    %% Demo demonstrates engine flow
    Demo -->|"demonstrates_flow"| Engine

    %% Testing Suite interactions
    TestsDir -->|"tests"| Providers
    TestsDir -->|"tests"| Serper
    TestsDir -->|"tests"| Engine

    %% Click Events
    click CLI "https://github.com/shibing624/deep-research/blob/main/main.py"
    click Gradio "https://github.com/shibing624/deep-research/blob/main/src/gradio_chat.py"
    click Engine "https://github.com/shibing624/deep-research/blob/main/src/deep_research.py"
    click ConfigModule "https://github.com/shibing624/deep-research/blob/main/src/config.py"
    click ConfigFile "https://github.com/shibing624/deep-research/blob/main/config.example.yaml"
    click Prompts "https://github.com/shibing624/deep-research/blob/main/src/prompts.py"
    click SearchUtils "https://github.com/shibing624/deep-research/blob/main/src/search_utils.py"
    click ModelUtils "https://github.com/shibing624/deep-research/blob/main/src/model_utils.py"
    click Providers "https://github.com/shibing624/deep-research/blob/main/src/providers.py"
    click Serper "https://github.com/shibing624/deep-research/blob/main/src/serper_client.py"
    click MP "https://github.com/shibing624/deep-research/blob/main/src/mp_search_client.py"
    click Tavily "https://github.com/shibing624/deep-research/blob/main/src/tavily_client.py"
    click Demo "https://github.com/shibing624/deep-research/blob/main/deep_research_demo.py"
    click Docs "https://github.com/shibing624/deep-research/blob/main/docs/project.md"
    click TestsDir "https://github.com/shibing624/deep-research/tree/main/tests/"
    click TestProviders "https://github.com/shibing624/deep-research/blob/main/tests/test_providers.py"
    click TestSerperTests "https://github.com/shibing624/deep-research/blob/main/tests/test_serper_client.py"

    %% Styles
    classDef userInterfaces fill:#bbdefb,stroke:#0d47a1,stroke-width:2px;
    classDef engine fill:#c8e6c9,stroke:#1b5e20,stroke-width:2px;
    classDef config fill:#ffccbc,stroke:#bf360c,stroke-width:2px;
    classDef prompt fill:#d1c4e9,stroke:#311b92,stroke-width:2px;
    classDef providers fill:#ffcdd2,stroke:#b71c1c,stroke-width:2px;
    classDef external fill:#ffe082,stroke:#f57f17,stroke-width:2px;
    classDef demo fill:#b2dfdb,stroke:#004d40,stroke-width:2px;
    classDef tests fill:#f8bbd0,stroke:#880e4f,stroke-width:2px;

    class CLI,Gradio userInterfaces
    class Engine engine
    class ConfigModule,ConfigFile config
    class Prompts,SearchUtils,ModelUtils prompt
    class Providers,Serper,MP,Tavily providers
    class ExtSerper,ExtMP,ExtTavily,ExtLLM external
    class Demo,Docs demo
    class TestsDir,TestProviders,TestSerperTests tests
```
