How fdarkaou/open-deep-research works
Main Components
• Frontend User Interface: Represented by the Next.js “app” directory and all UI components in the “components” directories (for chat, dialogs, buttons, etc.). These components render the interactive research interface, including real-time progress tracking and a markdown viewer.
• API Routes / Backend Endpoints: Found under “app/api”, with separate routes for research, feedback, and API-key management. These endpoints handle server-side logic, interacting with the research libraries and with external services.
• Deep Research Logic Library: Located in “lib/deep-research”, this encapsulates the core functionality such as generating search queries, managing recursive research, processing web scraping results, and assembling markdown reports.
• Auxiliary Libraries & Hooks: Located in “lib/hooks” and “lib/utils”, these support features like auto-scrolling and other utility functions.
• External Services: The system integrates with external APIs like OpenAI and FireCrawl. These services are crucial for natural language understanding and web scraping.
• Configuration & Environment: The environment files (e.g., .env.example, next.config.js) and static assets in “public” define deployment configurations, theming, and API key management (including secure HTTP-only cookies) through the API key dialog.
Relationships and Interactions
• The User (client) interacts with the Frontend UI, which is rendered through Next.js pages and components.
• Frontend components make API calls (via the defined API endpoints such as /api/research, /api/feedback, and /api/keys) to trigger deep research processes, manage API keys, and submit user feedback.
• The API endpoints then call upon the deep research library located in “lib/deep-research”. This library encapsulates the logic for generating search queries, splitting texts for AI processing, and managing research iterations.
• In turn, the deep research module communicates with external services (OpenAI for language modelling and FireCrawl for web search/scraping) to perform its operations.
• Responses (markdown reports, progress updates) are sent back through the API to the frontend, which then renders the results visually.
Architectural Patterns and Design Principles
• Full-Stack Separation: The clear division between frontend (Next.js pages and UI components) and backend (API routes and research libraries).
• Modular Design: Splitting the code into directories like “components”, “api”, and “lib” enables reuse and maintainability.
• API Key Management: Careful handling of sensitive keys via server-side storage (HTTP-only cookies) and a configurable API key dialog.
• Declarative UI with Component-Based Architecture: Visual consistency provided by shadcn/ui and custom components.
• Clear Separation of Concerns: The research business logic is encapsulated in “lib/deep-research”, separate from UI rendering and API facilitation.
Technologies, Frameworks, and Libraries
• Next.js (for server-side rendering, routing, and API handling)
• shadcn/ui (for UI components)
• Tailwind CSS (for styling, as indicated by tailwind.config.ts)
• Libraries in “lib/deep-research” that likely use custom AI or text-splitting algorithms
• External services: OpenAI and FireCrawl