# gerevai/gerev architecture

> Architecture diagram of the GitHub repository gerevai/gerev, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-31). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/gerevai/gerev
- Repository: https://github.com/gerevai/gerev

## Overview

##### Project Overview & Main Components
- This is a full-stack enterprise search engine built to ingest and search internal documents, conversations, and pages across various data sources (Slack, Confluence, Jira, Google Drive, etc.).
- It is composed of:
- A Frontend/UI layer (located in the "ui" directory) powered by a modern JavaScript framework (React with TypeScript, Tailwind CSS, and Webpack setup) to deliver a responsive search interface.
- A Backend/Application layer (in the "app" directory) written in Python. This layer handles API endpoints, business logic (search, indexing, data ingestion), and orchestrates interactions with external systems.
- A Data Integration/Source Module (within the "app/data_source" subdirectories) that contains connectors for various services (like Slack, Confluence, Jira, etc.), abstracting the ingestion of external data.
- An Indexing engine (in "app/indexing") which includes multiple strategies (BM25, FAISS) to build and search through indexes.
- A Background Processing/Worker system (e.g., the "app/queues" and "app/workers" directories) responsible for handling asynchronous tasks such as indexing and data processing.
- Database layer & Migrations (via Alembic scripts in "app/alembic") which is used for storing metadata, configuration, and possibly search indexes.

##### Business Flow & Interactions
- User Interaction:
- Users interact with the system via the Frontend. They enter search queries or manage configurations through browser-rendered pages.
- API Communication:
- The UI communicates with the Backend via RESTful API endpoints (illustrated in "app/api"). These endpoints handle tasks like processing search requests and managing data source configurations.
- Search Process:
- When a search query is submitted, the Backend’s search logic (found in modules such as "app/search_logic.py") receives the query, potentially consulting the Indexing engine to retrieve relevant results.
- Data Ingestion & Indexing:
- The system continually ingests data from various external data sources through the Data Source connectors. Each connector/module in "app/data_source/sources" fetches and formats data.
- Once data is ingested, background tasks (lifted from "app/queues" and "app/workers") are triggered. These tasks use the Indexing modules (BM25 and FAISS in "app/indexing") to update search indexes.
- Persistence:
- The database (managed with Alembic migrations) is used to persist configurations, indexing metadata, user information, and perhaps search logs.
- External Integrations:
- The architecture supports multiple external integrations. Each connector both fetches data from the external service and relays state/updates into the application’s indexing workflow.

##### Design Principles to Emphasize
- Modularity and Separation of Concerns: Clearly separate the UI from the backend, and the ingestion/indexing components from synchronous user request paths.
- Asynchronous Processing: Highlight how data ingestion and indexing are handled asynchronously, ensuring system scalability.
- Extensibility: Illustrate how additional data sources can be added through the “Data Integration Modules.”

##### Final Flow Chart Summary
- Data retrieval from the indexing engine for search results.
- Initiation of background tasks for data ingestion from external sources.
- Conclude with the results flowing back to the user, while also showing persistence in the database.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% User Interaction Layer
    subgraph "User Interaction"
        User["User"]:::client
        UIClient["UI / Client (React, Tailwind CSS)"]:::client
    end

    %% Backend Processing Layer
    subgraph "Backend Processing"
        API["Backend/API Layer"]:::backend
        SearchLogic["Search Logic (Python)"]:::backend
    end

    %% Data & Processing Layer
    subgraph "Processing Flow"
        DataModule["Data Integration Modules (Connectors)"]:::data
        BGProc["Background Processing/Worker System"]:::background
        IndexEngine["Indexing Engine (BM25, FAISS)"]:::indexing
    end

    %% Persistence Layer
    subgraph "Persistence"
        Database["Database & Migrations"]:::db
    end

    %% Connections
    User -->|"SendsQuery"| UIClient
    UIClient -->|"APIRequest"| API
    API -->|"ProcessQuery"| SearchLogic
    SearchLogic -->|"QueryIndex"| IndexEngine
    API -->|"DataIngestion"| DataModule
    DataModule -->|"TriggerTask"| BGProc
    BGProc -->|"IndexUpdate"| IndexEngine
    API -->|"ConfigurationSync"| Database
    API -->|"Response"| UIClient

    %% Click Events
    click UIClient "https://github.com/gerevai/gerev/tree/main/ui"
    click API "https://github.com/gerevai/gerev/tree/main/app/api"
    click DataModule "https://github.com/gerevai/gerev/tree/main/app/data_source"
    click IndexEngine "https://github.com/gerevai/gerev/tree/main/app/indexing"
    click BGProc "https://github.com/gerevai/gerev/tree/main/app/queues"
    click BGProc "https://github.com/gerevai/gerev/blob/main/app/workers.py"
    click Database "https://github.com/gerevai/gerev/tree/main/app/alembic"
    click SearchLogic "https://github.com/gerevai/gerev/blob/main/app/search_logic.py"

    %% Styles
    classDef client fill:#f1c232,stroke:#e69138,stroke-width:2px;
    classDef backend fill:#6aa84f,stroke:#38761d,stroke-width:2px;
    classDef data fill:#c9daf8,stroke:#1155cc,stroke-width:2px;
    classDef indexing fill:#b4a7d6,stroke:#6e5494,stroke-width:2px;
    classDef background fill:#f9cb9c,stroke:#e69138,stroke-width:2px;
    classDef db fill:#93c47d,stroke:#38761d,stroke-width:2px;
```
