How shibing624/deep-research works
Project Type and Purpose
• The project is an AI-powered research assistant written in Python. It performs iterative, deep research on any given topic by combining capabilities of search engine APIs (Serper, MP Search, Tavily) with large language models (LLM). This project supports multiple interfaces (command-line, Gradio web UI, and direct Python module integration) and is intended for both technical users and researchers.
• The README highlights features such as iterative search, automated query generation, multi-language support, and configurable clarification processes. It’s clear that the system is modular and designed to interact with various external services.
Analyzing the File Structure
• Top-level files and directories:
• main.py: Entry point for CLI commands, demo, and potentially starting the Gradio interface.
• deep_research_demo.py: A demo script showcasing a complete research flow.
• config.example.yaml: An example configuration file containing key settings (API keys, model hierarchies, search engine options).
• src directory: Contains core modules such as deep_research.py (main logic), config.py (configuration parsing), gradio_chat.py (interface for web UI), and various modules that interface with different search providers (serper_client.py, mp_search_client.py, tavily_client.py). Also includes model_utils, prompts, providers, and search_utils.
• tests directory: Contains unit tests for key components (providers and client integrations).
Main Components
a. User Interfaces and Entry Points:
• CLI (via main.py)
• Gradio Web Interface (triggered via main.py demo and handled by src/gradio_chat.py)
• Python Module usage (direct calls to src.deep_research and its asynchronous API)
b. Core Application Components:
• Deep Research Engine (src/deep_research.py): Responsible for managing the research flow, iteratively generating and refining queries, managing context, and calling the LLM.
• Configuration Management (src/config.py): Loads and manages configuration options (API keys, search preferences, context limits, etc.), typically fed from config.yaml.
• Provider Modules (src/providers.py alongside serper_client.py, mp_search_client.py, tavily_client.py): Encapsulate integrations with external search APIs.
• Prompt Generation and Utility Modules (src/prompts.py, src/search_utils.py, src/model_utils.py): Handle the creation of prompts for the LLM and processing of search results.
c. External Services:
• Search Engines: The API endpoints for Serper, MP Search, and Tavily.
• Potential LLM and Web scraping engines (even if only conceptually defined through the deep research flow).
d. Supporting Components:
• Demo and documentation (deep_research_demo.py and docs/ folder)
• Unit Testing Suite (tests/ directory)
Relationships and Data Flow
• This query is passed to the Deep Research Engine which:
• Reads configuration settings from the Config module.
• Generates initial prompts using the Prompt module.
• Dispatches queries to the appropriate search provider client based on configuration.
• Receives search results, applies search utility functions to refine or reformat results, and may invoke the LLM for deep context understanding.
• The outputs (final report, iterative research data) are then returned to the user via the chosen interface.
• Additionally, the testing components interact with both the provider clients and core modules to ensure system reliability.
Architectural Patterns and Technologies
• The architecture follows a modular design where each concern (search integration, prompt generation, configuration, user interface) is separated into its module under the src directory.
• The use of asynchronous programming (as seen in deep_research_stream) emphasizes non-blocking I/O when interacting with external APIs.
• Integration of a web UI using Gradio demonstrates a separation between backend research logic and frontend presentation.
• Built-in unit tests indicate a test-driven approach for quality assurance.