How ifitsmanu/ii-researcher works
Project Type and Purpose
• The project is a full-stack deep search agent that uses custom “BAML” functions to perform intelligent web searches, content scraping, multi-step reasoning, and answer generation. It integrates several pipelines (pipeline mode and reasoning mode) with external APIs and LLM integrations.
• The README highlights rich features such as asynchronous operations, configurable LLMs (through LiteLLM), and multiple search & scraping providers (like Tavily, SerpAPI, Firecrawl, etc). This indicates that the system is designed to combine web-based data retrieval with advanced AI reasoning.
Main Components
a. Frontend:
• Next.js application found in the "frontend" directory.
• Interaction with the backend API; annotate that it serves the user interface and communicates with API endpoints (e.g., http://localhost:8000).
b. Backend API:
• API service started with "api.py". This service serves as the bridge between the frontend and core processing logic.
• That it handles incoming requests (e.g., search queries) from the frontend and forwards them to the core pipeline.
c. Core II-Researcher Modules:
• Pipeline components including the action handler modules (action, reflect, search, visit) and the evaluator.
• Reasoning components with sub-modules for agent reasoning, clients (e.g., openai_client), models (action, output, trace), and tools (scrapers, compressors).
• Data flows between components like the “pipeline” and “reasoning” folders, which handle question processing, multi-step reasoning, and report generation.
d. BAML Integration:
• "baml_client" and "baml_src" directories. These are used for interfacing with BAML scripts and functions which structure the outputs. Note that “baml_client” contains both synchronous and asynchronous clients, tracing, and type utilities, while “baml_src” provides the domain-specific BAML scripts used in various tasks.
e. External LLM and API Integrations:
• An external LLM component (LiteLLM in this case) acting as a proxy to underlying AI models (OpenAI’s models, gpt-4o, etc.). It is configured via environment variables and launched either as a service through Docker or with a standalone CLI.
• Mention additional external providers: SerpAPI, Tavily, Firecrawl, which provide web search data and scraping functionalities.
• That these external services feed data into the reasoning and pipeline processing modules.
f. CLI Interface:
• "cli.py" module as a separate entry point that lets users run the system in two modes (pipeline mode and reasoning mode). Indicate that this communicates directly with core processing modules.
g. Docker Deployment:
• Docker components (docker-compose.yml, Dockerfiles under docker and frontend directories) as part of the overall deployment layer. This orchestrates container-based running for the frontend, API backend, and LiteLLM server.
Relationships and Data Flow
• From the user interface (frontend) → API server.
• API server routes requests to the core II-Researcher modules.
• Core modules interact with BAML clients and source scripts to process questions.
• Core processing sends requests to external services (search providers, LLMs via LiteLLM) and returns processed results.
• Data flow between CLI invocations and core reasoning components for both pipeline and reasoning modes.
• Link Docker containers together, showing how services (frontend, API, LiteLLM) communicate over defined ports.