How gerevai/gerev works
Project Overview & Main Components
• This is a full-stack enterprise search engine built to ingest and search internal documents, conversations, and pages across various data sources (Slack, Confluence, Jira, Google Drive, etc.).
• It is composed of:
• A Frontend/UI layer (located in the "ui" directory) powered by a modern JavaScript framework (React with TypeScript, Tailwind CSS, and Webpack setup) to deliver a responsive search interface.
• A Backend/Application layer (in the "app" directory) written in Python. This layer handles API endpoints, business logic (search, indexing, data ingestion), and orchestrates interactions with external systems.
• A Data Integration/Source Module (within the "app/data_source" subdirectories) that contains connectors for various services (like Slack, Confluence, Jira, etc.), abstracting the ingestion of external data.
• An Indexing engine (in "app/indexing") which includes multiple strategies (BM25, FAISS) to build and search through indexes.
• A Background Processing/Worker system (e.g., the "app/queues" and "app/workers" directories) responsible for handling asynchronous tasks such as indexing and data processing.
• Database layer & Migrations (via Alembic scripts in "app/alembic") which is used for storing metadata, configuration, and possibly search indexes.
Business Flow & Interactions
• User Interaction:
• Users interact with the system via the Frontend. They enter search queries or manage configurations through browser-rendered pages.
• API Communication:
• The UI communicates with the Backend via RESTful API endpoints (illustrated in "app/api"). These endpoints handle tasks like processing search requests and managing data source configurations.
• Search Process:
• When a search query is submitted, the Backend’s search logic (found in modules such as "app/search_logic.py") receives the query, potentially consulting the Indexing engine to retrieve relevant results.
• Data Ingestion & Indexing:
• The system continually ingests data from various external data sources through the Data Source connectors. Each connector/module in "app/data_source/sources" fetches and formats data.
• Once data is ingested, background tasks (lifted from "app/queues" and "app/workers") are triggered. These tasks use the Indexing modules (BM25 and FAISS in "app/indexing") to update search indexes.
• Persistence:
• The database (managed with Alembic migrations) is used to persist configurations, indexing metadata, user information, and perhaps search logs.
• External Integrations:
• The architecture supports multiple external integrations. Each connector both fetches data from the external service and relays state/updates into the application’s indexing workflow.
Design Principles to Emphasize
• Modularity and Separation of Concerns: Clearly separate the UI from the backend, and the ingestion/indexing components from synchronous user request paths.
• Asynchronous Processing: Highlight how data ingestion and indexing are handled asynchronously, ensuring system scalability.
• Extensibility: Illustrate how additional data sources can be added through the “Data Integration Modules.”
Final Flow Chart Summary
• Data retrieval from the indexing engine for search results.
• Initiation of background tasks for data ingestion from external sources.
• Conclude with the results flowing back to the user, while also showing persistence in the database.