How minh-ton/simple-deep-research works
Project Type & Purpose
• This is a research agent designed to perform “deep research” by interacting with LLMs (for follow-up questions, precise search queries, and evaluation) and external services (e.g., Firecrawl for search results).
• Its objective is to take a user’s research query, iteratively clarify and transform it into precise search queries, fetch and evaluate web content, and finally compile a research report. The approach is modular and task-based.
a. User Interface/Input:
– A starting point where the user provides the research query and possibly answers follow-up questions.
b. Core Agent/Workflow Orchestrator:
– Represented by the entry point (index.ts) that sets the research process in motion. This orchestrator coordinates multiple tasks.
c. Task Modules (located under src/tasks):
– “follow-up” task: Generates clarifying follow-up questions.
– “precise-query” task: Converts the user’s input into precise search queries.
– “generate-research” task: Combines step outputs into a final research report.
– “assess-sufficiency” task: Verifies if the fetched content addresses the research query.
– “extract-insights” task: Extracts key insights from the web page content.
d. Supporting Library Components (inside src/lib):
– “aisdk.ts”: Handles interactions with the OpenAI API.
– “firecrawl.ts”: Manages interactions with the Firecrawl API for Google searches.
e. External Services:
– OpenAI API for generating queries and evaluating content.
– Firecrawl API for conducting searches.
f. Data / Configuration:
– “constant-config.json” provides configuration parameters (controlled research depth, query limits, timeouts, etc.).
– Environment variables for API keys are configured using a .env file.
g. Output / Report:
– The final research report is output as a Markdown file (research/research.md).
Relationships & Data Flow
• The Workflow Orchestrator triggers:
• A “follow-up” step to get additional clarifying information from the LLM.
• A “precise-query” step to transform the input into refined queries.
• These queries are then used to call the Firecrawl API for a Google search. Search results are processed further.
• For each result:
• The “assess-sufficiency” module uses the LLM to decide if the content is relevant.
• The “extract-insights” module extracts the useful details.
• Finally, the “generate-research” module compiles the findings into a coherent research report.
Architectural Patterns & Design Principles
• The project employs a multi-step, pipeline-like architecture where each task/module focuses on a single responsibility.
• It makes use of modular design by dividing functionalities into clear, segregated tasks.
• External service interactions (with OpenAI and Firecrawl) are abstracted via dedicated libraries, reinforcing separation of concerns and ease of replacement or modification.
Recommendations for the Principal Software Engineer
• Components from “src/lib” to show how the system interfaces with external APIs.
• That any dependencies (for instance, using the “o3-mini” model on the OpenAI API or the specific settings from constant-config.json) are annotated to clarify behavior and configuration parameters.
• Avoid over-complicating; use as many distinct components as necessary so that every function (clarification, query generation, evaluation, extraction, final report) is clearly depicted.