# fdarkaou/open-deep-research architecture

> Architecture diagram of the GitHub repository fdarkaou/open-deep-research, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-02-14). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/fdarkaou/open-deep-research
- Repository: https://github.com/fdarkaou/open-deep-research

## Overview

##### Main Components
- Frontend User Interface: Represented by the Next.js “app” directory and all UI components in the “components” directories (for chat, dialogs, buttons, etc.). These components render the interactive research interface, including real-time progress tracking and a markdown viewer.
- API Routes / Backend Endpoints: Found under “app/api”, with separate routes for research, feedback, and API-key management. These endpoints handle server-side logic, interacting with the research libraries and with external services.
- Deep Research Logic Library: Located in “lib/deep-research”, this encapsulates the core functionality such as generating search queries, managing recursive research, processing web scraping results, and assembling markdown reports.
- Auxiliary Libraries & Hooks: Located in “lib/hooks” and “lib/utils”, these support features like auto-scrolling and other utility functions.
- External Services: The system integrates with external APIs like OpenAI and FireCrawl. These services are crucial for natural language understanding and web scraping.
- Configuration & Environment: The environment files (e.g., .env.example, next.config.js) and static assets in “public” define deployment configurations, theming, and API key management (including secure HTTP-only cookies) through the API key dialog.

##### Relationships and Interactions
- The User (client) interacts with the Frontend UI, which is rendered through Next.js pages and components.
- Frontend components make API calls (via the defined API endpoints such as /api/research, /api/feedback, and /api/keys) to trigger deep research processes, manage API keys, and submit user feedback.
- The API endpoints then call upon the deep research library located in “lib/deep-research”. This library encapsulates the logic for generating search queries, splitting texts for AI processing, and managing research iterations.
- In turn, the deep research module communicates with external services (OpenAI for language modelling and FireCrawl for web search/scraping) to perform its operations.
- Responses (markdown reports, progress updates) are sent back through the API to the frontend, which then renders the results visually.

##### Architectural Patterns and Design Principles
- Full-Stack Separation: The clear division between frontend (Next.js pages and UI components) and backend (API routes and research libraries).
- Modular Design: Splitting the code into directories like “components”, “api”, and “lib” enables reuse and maintainability.
- API Key Management: Careful handling of sensitive keys via server-side storage (HTTP-only cookies) and a configurable API key dialog.
- Declarative UI with Component-Based Architecture: Visual consistency provided by shadcn/ui and custom components.
- Clear Separation of Concerns: The research business logic is encapsulated in “lib/deep-research”, separate from UI rendering and API facilitation.

##### Technologies, Frameworks, and Libraries
- Next.js (for server-side rendering, routing, and API handling)
- shadcn/ui (for UI components)
- Tailwind CSS (for styling, as indicated by tailwind.config.ts)
- Libraries in “lib/deep-research” that likely use custom AI or text-splitting algorithms
- External services: OpenAI and FireCrawl

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Global User
    User["User"]:::external

    %% Frontend UI Subgraph
    subgraph "Frontend UI"
        A["Pages & Global Styles"]:::frontend
        B["Chat Components"]:::frontend
        C["UI Components"]:::frontend
    end

    %% API Endpoints Subgraph
    subgraph "API Endpoints"
        D["Research API"]:::backend
        E["Feedback API"]:::backend
        F["Keys API"]:::backend
    end

    %% Deep Research Library Subgraph
    subgraph "Deep Research Library"
        G["Core Research Logic"]:::logic
        H["AI & Text Processing"]:::logic
        G -->|"processes"| H
    end

    %% Auxiliary Libraries Subgraph
    subgraph "Auxiliary Libraries"
        I["Custom Hook"]:::auxiliary
        J["Utility Functions"]:::auxiliary
    end

    %% External Services Subgraph
    subgraph "External Services"
        K["OpenAI"]:::external
        L["FireCrawl"]:::external
    end

    %% Configuration & Environment Subgraph
    subgraph "Configuration & Environment"
        M["Environment File"]:::config
        N["Next.js Config"]:::config
        O["Tailwind CSS Config"]:::config
        P["Public Assets"]:::config
    end

    %% Main Interactions
    User -->|"accesses"| A
    User -->|"accesses"| B
    User -->|"accesses"| C

    A -->|"API_call"| D
    B -->|"API_call"| D
    C -->|"API_call"| D

    D -->|"invokes_logic"| G
    E -->|"sends_feedback"| A
    F -->|"key_exchange"| A

    G -->|"calls_OpenAI"| K
    G -->|"calls_FireCrawl"| L

    %% Auxiliary Libraries support Frontend
    A -.->|"utilize"| I
    B -.->|"utilize"| J

    %% Configuration access
    A --->|"reads"| M
    A --->|"reads"| N
    A --->|"reads"| O
    A --->|"reads"| P

    %% Legend subgraph
    subgraph "Legend"
        L1["Frontend UI = Blue"]
        L2["API Endpoints = Green"]
        L3["Business Logic = Yellow"]
        L4["Auxiliary Libraries = Orange"]
        L5["External Services = Red"]
        L6["Configuration = Purple"]
    end

    %% Click Events for Frontend UI
    click A "https://github.com/fdarkaou/open-deep-research/tree/main/app"
    click B "https://github.com/fdarkaou/open-deep-research/tree/main/components/chat"
    click C "https://github.com/fdarkaou/open-deep-research/tree/main/components/ui"

    %% Click Events for API Endpoints
    click D "https://github.com/fdarkaou/open-deep-research/blob/main/app/api/research/route.ts"
    click E "https://github.com/fdarkaou/open-deep-research/blob/main/app/api/feedback/route.ts"
    click F "https://github.com/fdarkaou/open-deep-research/blob/main/app/api/keys/route.ts"

    %% Click Events for Deep Research Library
    click G "https://github.com/fdarkaou/open-deep-research/tree/main/lib/deep-research"
    click H "https://github.com/fdarkaou/open-deep-research/tree/main/lib/deep-research/ai"

    %% Click Events for Auxiliary Libraries
    click I "https://github.com/fdarkaou/open-deep-research/blob/main/lib/hooks/use-scroll-to-bottom.ts"
    click J "https://github.com/fdarkaou/open-deep-research/blob/main/lib/utils.ts"

    %% Click Events for Configuration & Environment
    click M "https://github.com/fdarkaou/open-deep-research/blob/main/.env.example"
    click N "https://github.com/fdarkaou/open-deep-research/blob/main/next.config.js"
    click O "https://github.com/fdarkaou/open-deep-research/blob/main/tailwind.config.ts"
    click P "https://github.com/fdarkaou/open-deep-research/tree/main/public"

%% Class Definitions for Coloring
classDef frontend fill:#AED6F1,stroke:#1B4F72,stroke-width:2px;
classDef backend fill:#A9DFBF,stroke:#196F3D,stroke-width:2px;
classDef logic fill:#F9E79F,stroke:#9A7D0A,stroke-width:2px;
classDef auxiliary fill:#F5CBA7,stroke:#9C640C,stroke-width:2px;
classDef external fill:#F1948A,stroke:#922B21,stroke-width:2px;
classDef config fill:#D7BDE2,stroke:#6C3483,stroke-width:2px;
```
