How zilliztech/deep-searcher works
Main Components
• Configuration & Initialization: Represent the Configuration module (configuration.py, config.yaml) which sets up provider configurations for LLMs, Embeddings, vector databases, file loaders, and web crawlers.
• Core Processing Modules:
– Agent: Components under deepsearcher/agent (base, chain_of_rag, deep_search, etc.) that coordinate reasoning and searching.
– Embedding: Modules in deepsearcher/embedding that interface with various embedding models (OpenAIEmbedding, MilvusEmbedding, etc.).
– LLM: Components in deepsearcher/llm wrapping various language models (OpenAI, DeepSeek, Anthropic, etc.).
– Loader: The file loader and web crawler modules (in deepsearcher/loader) for ingesting local and online data.
– Vector Database: The deepsearcher/vector_db module that encapsulates interactions with Milvus, Oracle, Qdrant, etc.
• CLI & API Interface:
– CLI: The command-line interface (deepsearcher/cli.py) to load data and perform queries.
– FastAPI Service: main.py initiates a web service (accessible via localhost:8000/docs) providing API endpoints for query processing.
• Evaluation & Examples:
– Evaluation directory that supports performance assessment and result plotting.
– Examples directory that contains scripts demonstrating typical usage of the system.
Relationships and Interactions
• Configuration → [LLM, Embedding, VectorDB, File Loader, Web Crawler]: The configuration sets provider-specific parameters and is used across various modules.
• Data Ingestion Flow: The file loaders (file_loader) and web crawlers (web_crawler) import documents from local sources or online, which then get processed by the splitter and eventually indexed into the vector database.
• Query Handling Pipeline: Incoming queries (via CLI or FastAPI) are routed to the online_query module, where the agent module orchestrates calls to an LLM and an embedding service. The retrieved embeddings then interact with the VectorDB to search for relevant content.
• Output/Results: The agent compiles the results (e.g., reports or Q&A responses) that are returned to the end-user.
Architectural Patterns & Design Principles
• Modular Design: Separate modules for embeddings, LLMs, loaders, and vector databases, which promotes plug-and-play extensibility.
• Provider Abstraction: Each category (LLM, embedding, file loader, etc.) is abstracted behind a configuration interface, making it extensible for multiple external providers.
• API and CLI Exposure: Dual interface mode (CLI and web service) allows for flexibility in integration.
• Support for both offline and online modes for data ingestion and query processing.
Relevant Technologies and Frameworks
• Python (3.10+)
• FastAPI for the web service (main.py)
• CLI scripting for command-line operations.
• Various third-party APIs and libraries for LLMs (OpenAI, DeepSeek, Anthropic, etc.), embedding models, and vector databases (Milvus, Qdrant, Oracle).
Final Checklist for the Principal Software Engineer
• Every module (Configuration, Agents, Embedding, LLM, Loader, VectorDB, CLI/API) is clearly labeled.
– Configuration to each service (LLM, Embedding, etc.)
– Data flow from ingestion (file loaders/web crawlers) to vector database.
– Query requests from CLI/API to the agent.
• Critical control flows like initialization (via main.py FastAPI service) and query handling.