How activeinferenceinstitute/research-discovery-engine works
Project Type and Purpose
• From the README, IntelliDE (Research Discovery Engine) is clearly an AI-powered, interdisciplinary research platform focused on processing scientific literature, extracting structured knowledge, generating hypotheses, and visualizing complex knowledge maps.
• The system leverages advanced NLP techniques, vector-based representations, and even concepts like CT-GIN (Category Theory and Graph Isomorphism Networks) and active inference.
• The project is a full-stack application with both web interfaces and Python-based backend processing pipelines.
File Structure
• Notice that the repository is split into at least two major segments:
• The “resnei” directory contains Python files and related templates. This appears to be responsible for core backend functionalities such as document processing, markdown rendering, or early iterations of the processing pipeline.
• The “website_explore_the_unknown” directory is clearly a modern web application (using Next.js and TypeScript) exhibiting API routes (in Python for some endpoints, in TypeScript for others) and various frontend pages for chat, discovery, team, and waitlist. This likely represents the user interface and API layer.
• Also present are configuration files for GitHub issue templates (under .github) and build/deployment (e.g., next.config.js, package.json) that indicate an active development process.
Key Components
a. Frontend Layer:
• Next.js web application (from “website_explore_the_unknown/app”) where users interact with the system. This layer includes pages such as:
• /chat (chat-interface for user interactions)
• /discover (visualizations for knowledge maps or discovery insights)
• /team & /waitlist (collaboration and user management features )
• General UI components (detailed under “components” and “components/ui”)
b. Backend/API Layer:
• The API server present in “website_explore_the_unknown/api” that includes endpoints for chat, subscription, and Flask-based test routes. These API endpoints serve as the intermediary between the frontend and backend processing.
• The Python scripts in the “resnei” folder likely provide additional backend functionalities such as document processing (e.g., “app.py”, “markdown_renderer.py”) and may constitute part of the data extraction or preprocessing pipelines.
c. Core Scientific Processing Engine:
• Technologies supporting this core include advanced NLP (BERTopic, SciBERT), vector-based knowledge representation, and custom graph theoretical algorithms (CT-GIN Framework).
d. External Integrations/Services:
• While not explicitly represented in the file tree, the README mentions external services such as literature sources, knowledge databases, or even API endpoints for third-party integration (e.g., links to the Active Inference Institute or research paper resources).
Architectural Patterns and Design Principles
• The design clearly distinguishes between the “presentation/UI” layer (Next.js-based frontend) and the “business logic” / “data processing” layer (Python-based document processing engine).
• Separation of concerns: Keep the data extraction/processing (using advanced AI/NLP) separate from the interaction layer that users see.
• Modular design: Each subdirectory (e.g., “api”, “app”, “resnei”) represents discrete modules that can be diagrammed as separate components communicating together.