# sentient-agi/opendeepsearch architecture

> Architecture diagram of the GitHub repository sentient-agi/opendeepsearch, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-03). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/sentient-agi/opendeepsearch
- Repository: https://github.com/sentient-agi/opendeepsearch

## Overview

##### Main Architectural Components
- Core Library (src/opendeepsearch): This is the backbone of the project and can be broken down into several sub-components:
- Context Building: Modules like build_context.py and process_sources_pro.py that gather and synthesize search context.
- Context Scraping: Folder “context_scraping” includes scrapers (basic_web_scraper.py, fast_scraper.py, crawl4ai_scraper.py) which handle web crawling and data extraction.
- Ranking Models: Within “ranking_models”, components such as base_reranker.py, chunker.py, infinity_rerank.py, and jina_reranker.py implement and manage semantic reranking of search results.
- Additional Tools: Modules such as ods_agent.py, ods_tool.py, prompts.py, and wolfram_tool.py, often used in combination with external packages, integrate with external tools like WolframAlpha and support agent-based interactions.

- API & External Integrations:
- Search Providers: External services such as Serper.dev or SearXNG are used to perform initial searches.
- LLM and Model Providers: Integration with Hugging Face, LiteLLM, OpenRouter, and potentially other LLM platforms.
- External Evaluation: The “evals” directory contains scripts and datasets for testing and evaluating search modes and performance.

- User Interfaces:
- A Gradio demo (gradio_demo.py) provides a graphical interface that a user can interact with.
- Optional command-line interactions where API keys and parameters are set via environment variables (.env.example).

- Configurations & Environment:
- Deployment and dependency management files (pyproject.toml, pdm.lock, requirements.txt) indicate how the application is built and deployed.
- The .env.example file signals that runtime parameters (like API keys) are externally configured.

##### Component Interactions and Data Flow
- User Interaction Layer:
- Users input a query via a CLI or through the Gradio web interface.
- The query is routed to the core search tool (OpenDeepSearchTool).

- Search Request Handling:
- The tool selects a search provider (using API keys provided via environment variables) to perform the initial search.
- Data flows from external search providers into the core.

- Context Scraping & Building:
- The scraping modules fetch additional content from web pages.
- The context-building modules then aggregate, format, and pass this data along to the ranking models.

- Semantic Reranking:
- The ranking models (Jina, Infinity, etc.) process the gathered results and re-rank them based on semantic similarity, utilizing chosen LLMs.

- Integration with External AI Agents:
- Modules such as ods_agent.py and integrations with SmolAgents & LiteLLM illustrate how the tool can act as a support agent in an orchestration context, making use of additional AI services.

- Evaluation and Testing:
- The evals directory provides scripts that test different aspects of retrieval and ranking, ensuring that each component performs as expected.

##### Architectural Patterns and Technologies
- Separation of Concerns: The design is modular—with distinct directories for scraping, context processing, ranking, and integration—allowing each module to be developed, tested, and maintained independently.
- Extensibility & Configurability: By depending on environment variables and external configuration files, the architecture supports easy integration with various API providers and models.
- Microservices/Plugin-like Structure: Some components (like external search providers or ranking models) can be swapped out or extended independently.
- API Layer: The use of clear interfaces (for example, through agent tools and consistent APIs for search and ranking) simplifies integration with other systems such as SmolAgents.

## Diagram (Mermaid)

```mermaid
graph TD
  %% User Interaction Layer
  UI["User Interface (Gradio Demo)"]:::interface
  SRM["Search Request Manager"]:::internal

  %% External Integrations
  ESP["External Search Provider (Serp Search)"]:::external

  %% Core Library and its submodules
  subgraph "Core Library"
      CL["Core Library"]:::internal
      CS["Context Scraping Module"]:::internal
      CB["Context Building Module"]:::internal
      RM["Ranking Models"]:::internal
      
      subgraph "Agent & Tool Integrations"
          OA["ods_agent"]:::internal
          OT["ods_tool"]:::internal
          PP["prompts"]:::internal
          WT["Wolfram Tool"]:::internal
      end
  end

  %% Evaluation and Configuration
  ET["Evaluation & Testing"]:::internal
  CONF["Configurations & Environment"]:::config

  %% Data Flow
  UI -->|"UserQuery"| SRM
  SRM -->|"APIRequest"| ESP
  ESP -->|"SearchResults"| SRM
  SRM -->|"ForwardData"| CS
  CS -->|"ExtractedData"| CB
  CB -->|"AggregatedContext"| RM
  RM -->|"SemanticRerank"| SRM
  SRM -->|"AgentIntegration"| OA
  SRM -->|"AgentIntegration"| OT
  SRM -->|"AgentIntegration"| PP
  SRM -->|"AgentIntegration"| WT
  SRM -->|"PerformanceData"| ET
  CONF -->|"ProvidesConfig"| SRM

  %% Internal Component Relationships
  CL --> CS
  CL --> CB
  CL --> RM
  CL --> OA
  CL --> OT
  CL --> PP
  CL --> WT

  %% Click Events
  click CL "https://github.com/sentient-agi/opendeepsearch/tree/main/src/opendeepsearch"
  click CB "https://github.com/sentient-agi/opendeepsearch/tree/main/src/opendeepsearch/context_building"
  click CS "https://github.com/sentient-agi/opendeepsearch/tree/main/src/opendeepsearch/context_scraping"
  click RM "https://github.com/sentient-agi/opendeepsearch/tree/main/src/opendeepsearch/ranking_models"
  click OA "https://github.com/sentient-agi/opendeepsearch/blob/main/src/opendeepsearch/ods_agent.py"
  click OT "https://github.com/sentient-agi/opendeepsearch/blob/main/src/opendeepsearch/ods_tool.py"
  click PP "https://github.com/sentient-agi/opendeepsearch/blob/main/src/opendeepsearch/prompts.py"
  click WT "https://github.com/sentient-agi/opendeepsearch/blob/main/src/opendeepsearch/wolfram_tool.py"
  click ESP "https://github.com/sentient-agi/opendeepsearch/tree/main/src/opendeepsearch/serp_search"
  click ET "https://github.com/sentient-agi/opendeepsearch/tree/main/evals"
  click UI "https://github.com/sentient-agi/opendeepsearch/blob/main/gradio_demo.py"
  click CONF "https://github.com/sentient-agi/opendeepsearch/blob/main/.env.example"

  %% Styles
  classDef internal fill:#a0d468,stroke:#4b8b1f,stroke-width:2px;
  classDef external fill:#ed5565,stroke:#b52b33,stroke-width:2px;
  classDef interface fill:#4fc1e9,stroke:#3a9fd9,stroke-width:2px;
  classDef config fill:#ffce54,stroke:#f6bb42,stroke-width:2px;
```
