# activeinferenceinstitute/research-discovery-engine architecture

> Architecture diagram of the GitHub repository activeinferenceinstitute/research-discovery-engine, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/activeinferenceinstitute/research-discovery-engine
- Repository: https://github.com/activeinferenceinstitute/research-discovery-engine

## Overview

##### Project Type and Purpose
- From the README, IntelliDE (Research Discovery Engine) is clearly an AI-powered, interdisciplinary research platform focused on processing scientific literature, extracting structured knowledge, generating hypotheses, and visualizing complex knowledge maps.
- The system leverages advanced NLP techniques, vector-based representations, and even concepts like CT-GIN (Category Theory and Graph Isomorphism Networks) and active inference.
- The project is a full-stack application with both web interfaces and Python-based backend processing pipelines.

##### File Structure
- Notice that the repository is split into at least two major segments:
- The “resnei” directory contains Python files and related templates. This appears to be responsible for core backend functionalities such as document processing, markdown rendering, or early iterations of the processing pipeline.
- The “website_explore_the_unknown” directory is clearly a modern web application (using Next.js and TypeScript) exhibiting API routes (in Python for some endpoints, in TypeScript for others) and various frontend pages for chat, discovery, team, and waitlist. This likely represents the user interface and API layer.
- Also present are configuration files for GitHub issue templates (under .github) and build/deployment (e.g., next.config.js, package.json) that indicate an active development process.

##### Key Components
a. Frontend Layer:
- Next.js web application (from “website_explore_the_unknown/app”) where users interact with the system. This layer includes pages such as:
- /chat (chat-interface for user interactions)
- /discover (visualizations for knowledge maps or discovery insights)
- /team & /waitlist (collaboration and user management features )
- General UI components (detailed under “components” and “components/ui”)
b. Backend/API Layer:
- The API server present in “website_explore_the_unknown/api” that includes endpoints for chat, subscription, and Flask-based test routes. These API endpoints serve as the intermediary between the frontend and backend processing.
- The Python scripts in the “resnei” folder likely provide additional backend functionalities such as document processing (e.g., “app.py”, “markdown_renderer.py”) and may constitute part of the data extraction or preprocessing pipelines.
c. Core Scientific Processing Engine:
- Technologies supporting this core include advanced NLP (BERTopic, SciBERT), vector-based knowledge representation, and custom graph theoretical algorithms (CT-GIN Framework).
d. External Integrations/Services:
- While not explicitly represented in the file tree, the README mentions external services such as literature sources, knowledge databases, or even API endpoints for third-party integration (e.g., links to the Active Inference Institute or research paper resources).

##### Architectural Patterns and Design Principles
- The design clearly distinguishes between the “presentation/UI” layer (Next.js-based frontend) and the “business logic” / “data processing” layer (Python-based document processing engine).
- Separation of concerns: Keep the data extraction/processing (using advanced AI/NLP) separate from the interaction layer that users see.
- Modular design: Each subdirectory (e.g., “api”, “app”, “resnei”) represents discrete modules that can be diagrammed as separate components communicating together.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Frontend Layer
    subgraph "Frontend Layer"
        UI["Next.js Frontend (UI Layer)"]:::frontend
    end

    %% API Layer
    subgraph "API Layer"
        API1["API Endpoints (Python)"]:::api
        API2["API Endpoints (TypeScript)"]:::api
    end

    %% Processing Layer
    subgraph "Processing Layer"
        ProcApp["Document Processing (app.py)"]:::processing
        Markdown["Markdown Renderer"]:::processing
    end

    %% Legacy UI Layer
    subgraph "Legacy UI"
        Templates["Template Rendering (HTML Views)"]:::legacy
    end

    %% External Services
    subgraph "External Services"
        External["External Data Sources & Third-Party Services"]:::external
    end

    %% Connections
    UI -->|"calls"| API1
    UI -->|"calls"| API2

    API1 -->|"processes"| ProcApp
    API1 -->|"renders"| Markdown
    API1 -->|"legacyCall"| Templates

    API2 -->|"processes"| ProcApp
    API2 -->|"renders"| Markdown

    ProcApp -->|"response"| UI
    Markdown -->|"response"| UI
    Templates -->|"response"| UI

    External -->|"feedsData"| ProcApp
    External -->|"provides"| UI

    %% Click Events
    click UI "https://github.com/activeinferenceinstitute/research-discovery-engine/tree/main/website_explore_the_unknown/app"
    click API1 "https://github.com/activeinferenceinstitute/research-discovery-engine/tree/main/website_explore_the_unknown/api"
    click API2 "https://github.com/activeinferenceinstitute/research-discovery-engine/tree/main/website_explore_the_unknown/app/api"
    click ProcApp "https://github.com/activeinferenceinstitute/research-discovery-engine/blob/main/resnei/app.py"
    click Markdown "https://github.com/activeinferenceinstitute/research-discovery-engine/blob/main/resnei/uploads/markdown_renderer.py"
    click Markdown "https://github.com/activeinferenceinstitute/research-discovery-engine/blob/main/resnei/resnei/uploads/markdown_renderer.py"
    click Templates "https://github.com/activeinferenceinstitute/research-discovery-engine/tree/main/resnei/resnei/templates"

    %% Styles
    classDef frontend fill:#AED6F1,stroke:#1F618D,stroke-width:2px;
    classDef api fill:#A3E4D7,stroke:#117864,stroke-width:2px;
    classDef processing fill:#F9E79F,stroke:#B7950B,stroke-width:2px;
    classDef legacy fill:#F5CBA7,stroke:#D35400,stroke-width:2px;
    classDef external fill:#E6B0AA,stroke:#922B21,stroke-width:2px;
```
