How sentient-agi/opendeepsearch works
Main Architectural Components
• Core Library (src/opendeepsearch): This is the backbone of the project and can be broken down into several sub-components:
• Context Building: Modules like build_context.py and process_sources_pro.py that gather and synthesize search context.
• Context Scraping: Folder “context_scraping” includes scrapers (basic_web_scraper.py, fast_scraper.py, crawl4ai_scraper.py) which handle web crawling and data extraction.
• Ranking Models: Within “ranking_models”, components such as base_reranker.py, chunker.py, infinity_rerank.py, and jina_reranker.py implement and manage semantic reranking of search results.
• Additional Tools: Modules such as ods_agent.py, ods_tool.py, prompts.py, and wolfram_tool.py, often used in combination with external packages, integrate with external tools like WolframAlpha and support agent-based interactions.
• API & External Integrations:
• Search Providers: External services such as Serper.dev or SearXNG are used to perform initial searches.
• LLM and Model Providers: Integration with Hugging Face, LiteLLM, OpenRouter, and potentially other LLM platforms.
• External Evaluation: The “evals” directory contains scripts and datasets for testing and evaluating search modes and performance.
• User Interfaces:
• A Gradio demo (gradio_demo.py) provides a graphical interface that a user can interact with.
• Optional command-line interactions where API keys and parameters are set via environment variables (.env.example).
• Configurations & Environment:
• Deployment and dependency management files (pyproject.toml, pdm.lock, requirements.txt) indicate how the application is built and deployed.
• The .env.example file signals that runtime parameters (like API keys) are externally configured.
Component Interactions and Data Flow
• User Interaction Layer:
• Users input a query via a CLI or through the Gradio web interface.
• The query is routed to the core search tool (OpenDeepSearchTool).
• Search Request Handling:
• The tool selects a search provider (using API keys provided via environment variables) to perform the initial search.
• Data flows from external search providers into the core.
• Context Scraping & Building:
• The scraping modules fetch additional content from web pages.
• The context-building modules then aggregate, format, and pass this data along to the ranking models.
• Semantic Reranking:
• The ranking models (Jina, Infinity, etc.) process the gathered results and re-rank them based on semantic similarity, utilizing chosen LLMs.
• Integration with External AI Agents:
• Modules such as ods_agent.py and integrations with SmolAgents & LiteLLM illustrate how the tool can act as a support agent in an orchestration context, making use of additional AI services.
• Evaluation and Testing:
• The evals directory provides scripts that test different aspects of retrieval and ranking, ensuring that each component performs as expected.
Architectural Patterns and Technologies
• Separation of Concerns: The design is modular—with distinct directories for scraping, context processing, ranking, and integration—allowing each module to be developed, tested, and maintained independently.
• Extensibility & Configurability: By depending on environment variables and external configuration files, the architecture supports easy integration with various API providers and models.
• Microservices/Plugin-like Structure: Some components (like external search providers or ranking models) can be swapped out or extended independently.
• API Layer: The use of clear interfaces (for example, through agent tools and consistent APIs for search and ranking) simplifies integration with other systems such as SmolAgents.