# vishnusharma03/pipeshub-ai architecture

> Architecture diagram of the GitHub repository vishnusharma03/pipeshub-ai, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-08-13). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/vishnusharma03/pipeshub-ai
- Repository: https://github.com/vishnusharma03/pipeshub-ai

## Overview

##### Project Type and Purpose
- This is a full-stack, modular, microservices-style Workplace AI platform.
- Frontend: React + TypeScript (Vite), renders UI for search, configuration, dashboards.
- Backend Node.js: REST API gateway, authentication, configuration management, crawling manager, enterprise search, notification, storage, tokens, user management modules.
- Backend Python: data ingestion, connectors (Google, Atlassian, Microsoft, S3, local KB, etc.), indexing pipeline, search services, AI agents, Celery tasks, messaging consumers/producers.
- Infrastructure: Docker Compose deployment, message broker (Kafka), databases (ArangoDB, MongoDB, Redis), vector DB (Qdrant), key-value store (etcd), Prometheus metrics.

##### File Structure Analysis
- Top-level:
- frontend/ (React UI)
- backend/nodejs/apps (gateway API service)
- backend/python/app (ingestion, connectors, indexing, search)
- deployment/docker-compose (dev/prod compose files)
- Patterns:
- Modular code per domain (auth, storage, crawling, enterprise_search, knowledge_base, etc.).
- Layered structure: controllers → services → repositories/providers → utils.
- Hexagonal/ports-and-adapters: keyValueStoreFactory, messaging factory, graph_db_factory, vector_db_factory.
- Messaging: Kafka consumers/producers for asynchronous events.
- Scheduler: Celery + Redis scheduler.
- CI/CD: GitHub workflows, Dockerfiles.

##### Key README Insights
- Frontend ↔ API Gateway (Node.js)
- Connector Services (Python) → Message Broker → Indexer/Searcher
- Datastores: Knowledge Graph (ArangoDB), Document DB (MongoDB), KV Store (etcd/Redis), Vector DB (Qdrant)
- Observability: Prometheus
- External Services/APIs: Google, Microsoft, Slack, Jira, Confluence, etc.
- Deployment via Docker Compose for cloud or on-prem.

- Frontend (React UI)
- API Gateway (Node.js service)
- Connector Workers (Python processes/agents)
- Asynchronous Message Broker (Kafka)
- Indexing & Search Engine (Python indexing service, Celery tasks)
- Datastores:
– Knowledge Graph DB (ArangoDB)
– Document Store (MongoDB)
– Key-Value Store (etcd/Redis)
– Vector DB (Qdrant)
- External Connectors (Google Workspace, Microsoft 365, Slack, Jira, Confluence, S3, local KB)
- Observability (Prometheus)

1. User browser → Frontend → API Gateway (HTTP/REST, JWT/SAML auth)
2. API Gateway ↔ Datastores (ArangoDB, MongoDB, etc.) for runtime queries and configuration
3. API Gateway → publishes events → Kafka topics (e.g., connect, sync, token)
4. Connector Workers subscribe to Kafka, pull data from External Services, transform and push to Indexer
5. Indexer (Celery tasks) writes processed documents/entities to ArangoDB, MongoDB, Qdrant
6. Search API queries ArangoDB and Vector DB, returns ranked results with citations
7. Observability metrics piped to Prometheus

c. Architectural Patterns & Principles:
- Microservices & Modularization: Each domain module is loosely coupled, independently deployable.
- Hexagonal Architecture: Adapters for connectors, stores, messaging.
- Event-Driven: Kafka for decoupling ingestion, sync, and indexing.
- Asynchronous Processing: Celery + Redis for long-running jobs (OCR, PDF parsing, AI model inference).
- No-Code App Builders: modular “builders” in Python for custom workflows.
- Security: OAuth, JWT, SAML, RBAC middlewares.

d. Relevant Technologies & Frameworks:
- Frontend: React, TypeScript, Vite, MUI, Zustand/Context API
- API Gateway: Node.js, TypeScript, Express, InversifyJS (DI container), Prometheus instrumentation, Kafka client, ArangoJS, Mongoose, etc.
- Ingestion/Search: Python, FastAPI, Celery, ai_arango_service, Qdrant, Pymupdf, docling, OpenAI SDK (or Ollama), ArangoDB Python driver
- Messaging: Kafka (brokers, topics, producers/consumers)
- Datastores: ArangoDB (graph), MongoDB (documents), Redis (KV cache, scheduler), etcd (config store), Qdrant (vector embeddings)
- Deployment: Docker, Docker Compose, GitHub Actions CI

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Layers Top to Bottom
    subgraph "User Layer"
        direction TB
        Browser["User Browser"]:::external
    end

    subgraph "Frontend Layer"
        direction TB
        Frontend["React UI"]:::service
    end

    subgraph "API Layer"
        direction TB
        APIGateway["API Gateway (Node.js)"]:::service

        subgraph "Gateway Services"
            direction TB
            AuthService["Auth Service"]:::service
            ConfigService["Configuration Service"]:::service
            CrawlingService["Crawling Manager"]:::service
            SearchService["Enterprise Search"]:::service
            KBService["Knowledge Base"]:::service
            StorageService["Storage Service"]:::service
            NotificationService["Notification Service"]:::service
            TokenService["Tokens Manager"]:::service
            UserService["User Management"]:::service
        end

        subgraph "Gateway Libraries"
            direction TB
            KVFactory["KeyValueStore Factory"]:::service
            KafkaService["Messaging (Kafka)"]:::service
            PrometheusLib["Prometheus Instrumentation"]:::service
        end
    end

    subgraph "Message Bus"
        direction TB
        Kafka["Kafka Broker"]:::broker
    end

    subgraph "Data Processing Layer"
        direction TB
        PythonAPI["Python REST API"]:::service
        ConnectorWorkers["Connector Workers"]:::service
        AIAgents["Python Agents & AI Actions"]:::service
        Indexer["Indexing Pipeline (Celery)"]:::service
    end

    subgraph "Data Stores"
        direction TB
        ArangoDB["ArangoDB (Graph DB)"]:::database
        MongoDB["MongoDB (Document DB)"]:::database
        KVStore["Redis/etcd (KV Store)"]:::database
        VectorDB["Qdrant (Vector DB)"]:::database
    end

    subgraph "External Systems"
        direction TB
        Google["Google Workspace"]:::external
        Microsoft["Microsoft 365"]:::external
        Slack["Slack"]:::external
        Jira["Jira/Confluence"]:::external
        S3["S3/Local KB"]:::external
    end

    subgraph "Observability"
        direction TB
        Prometheus["Prometheus"]:::observability
    end

    %% Connections
    Browser -->|"HTTP/REST, JWT/SAML"| Frontend
    Frontend -->|"HTTP/REST"| APIGateway

    APIGateway -->|"manage auth/config"| AuthService
    APIGateway -->|"manage config"| ConfigService
    APIGateway -->|"manage crawling"| CrawlingService
    APIGateway -->|"manage search"| SearchService
    APIGateway -->|"manage KB"| KBService
    APIGateway -->|"manage storage"| StorageService
    APIGateway -->|"produce notifications"| NotificationService
    APIGateway -->|"manage tokens"| TokenService
    APIGateway -->|"manage users"| UserService

    APIGateway --> KVFactory
    APIGateway --> KafkaService
    APIGateway --> PrometheusLib

    APIGateway -->|"publish events (Kafka)"| Kafka
    Kafka -->|"subscribe"| ConnectorWorkers
    Kafka -->|"subscribe"| Indexer

    ConnectorWorkers -->|"call external APIs"| Google
    ConnectorWorkers -->|"call external APIs"| Microsoft
    ConnectorWorkers -->|"call external APIs"| Slack
    ConnectorWorkers -->|"call external APIs"| Jira
    ConnectorWorkers -->|"fetch from"| S3
    ConnectorWorkers -->|"send to indexer"| Indexer

    Indexer -->|"write entities"| ArangoDB
    Indexer -->|"write docs"| MongoDB
    Indexer -->|"write embeddings"| VectorDB
    Indexer -->|"cache/store"| KVStore

    SearchService -->|"query"| ArangoDB
    SearchService -->|"vector search"| VectorDB

    APIGateway -->|"runtime queries"| ArangoDB
    APIGateway -->|"runtime queries"| MongoDB
    APIGateway -->|"runtime queries"| KVStore

    APIGateway --> Prometheus
    ConnectorWorkers --> Prometheus
    Indexer --> Prometheus

    %% Click Events
    click Frontend "https://github.com/vishnusharma03/pipeshub-ai/tree/main/frontend/"
    click APIGateway "https://github.com/vishnusharma03/pipeshub-ai/blob/main/backend/nodejs/apps/src/app.ts"
    click APIGateway "https://github.com/vishnusharma03/pipeshub-ai/blob/main/backend/nodejs/apps/src/index.ts"
    click AuthService "https://github.com/vishnusharma03/pipeshub-ai/tree/main/backend/nodejs/apps/src/modules/auth"
    click ConfigService "https://github.com/vishnusharma03/pipeshub-ai/tree/main/backend/nodejs/apps/src/modules/configuration_manager"
    click CrawlingService "https://github.com/vishnusharma03/pipeshub-ai/tree/main/backend/nodejs/apps/src/modules/crawling_manager"
    click SearchService "https://github.com/vishnusharma03/pipeshub-ai/tree/main/backend/nodejs/apps/src/modules/enterprise_search"
    click KBService "https://github.com/vishnusharma03/pipeshub-ai/tree/main/backend/nodejs/apps/src/modules/knowledge_base"
    click StorageService "https://github.com/vishnusharma03/pipeshub-ai/tree/main/backend/nodejs/apps/src/modules/storage"
    click NotificationService "https://github.com/vishnusharma03/pipeshub-ai/tree/main/backend/nodejs/apps/src/modules/notification"
    click TokenService "https://github.com/vishnusharma03/pipeshub-ai/tree/main/backend/nodejs/apps/src/modules/tokens_manager"
    click UserService "https://github.com/vishnusharma03/pipeshub-ai/tree/main/backend/nodejs/apps/src/modules/user_management"
    click KVFactory "https://github.com/vishnusharma03/pipeshub-ai/blob/main/backend/nodejs/apps/src/libs/keyValueStore/keyValueStoreFactory.ts"
    click KVFactory "https://github.com/vishnusharma03/pipeshub-ai/tree/main/backend/nodejs/apps/src/libs/keyValueStore/providers"
    click KafkaService "https://github.com/vishnusharma03/pipeshub-ai/blob/main/backend/nodejs/apps/src/libs/services/kafka.service.ts"
    click KafkaService "https://github.com/vishnusharma03/pipeshub-ai/blob/main/backend/nodejs/apps/src/libs/services/keyValueStore.service.ts"
    click PrometheusLib "https://github.com/vishnusharma03/pipeshub-ai/blob/main/backend/nodejs/apps/src/libs/services/prometheus/prometheus.service.ts"
    click PrometheusLib "https://github.com/vishnusharma03/pipeshub-ai/blob/main/backend/nodejs/apps/src/libs/middlewares/prometheus.middleware.ts"
    click ConnectorWorkers "https://github.com/vishnusharma03/pipeshub-ai/tree/main/backend/python/app/connectors"
    click AIAgents "https://github.com/vishnusharma03/pipeshub-ai/tree/main/backend/python/app/agents"
    click PythonAPI "https://github.com/vishnusharma03/pipeshub-ai/tree/main/backend/python/app/api"
    click Indexer "https://github.com/vishnusharma03/pipeshub-ai/blob/main/backend/python/app/core/celery_app.py"
    click Indexer "https://github.com/vishnusharma03/pipeshub-ai/blob/main/backend/python/app/modules/indexing/run.py"
    click Parser "https://github.com/vishnusharma03/pipeshub-ai/tree/main/backend/python/app/modules/parsers"
    click AIAgents "https://github.com/vishnusharma03/pipeshub-ai/tree/main/backend/python/app/modules/agents"
    click AIAgents "https://github.com/vishnusharma03/pipeshub-ai/blob/main/backend/python/app/agents/router/router.py"
    click ArangoDB "https://github.com/vishnusharma03/pipeshub-ai/blob/main/backend/python/app/services/graph_db/arango/arango.py"
    click ArangoDB "https://github.com/vishnusharma03/pipeshub-ai/blob/main/backend/python/app/services/graph_db/graph_db_factory.py"
    click VectorDB "https://github.com/vishnusharma03/pipeshub-ai/blob/main/backend/python/app/services/vector_db/qdrant/qdrant.py"
    click VectorDB "https://github.com/vishnusharma03/pipeshub-ai/blob/main/backend/python/app/services/vector_db/vector_db_factory.py"
    click KVStore "https://github.com/vishnusharma03/pipeshub-ai/blob/main/backend/python/app/services/key_value/redis/redis.py"
    click KVStore "https://github.com/vishnusharma03/pipeshub-ai/blob/main/backend/python/app/services/key_value/key_value_factory.py"
    click Messaging "https://github.com/vishnusharma03/pipeshub-ai/tree/main/backend/python/app/services/messaging/kafka"
    click Messaging "https://github.com/vishnusharma03/pipeshub-ai/blob/main/backend/python/app/services/messaging/messaging_factory.py"
    click Scheduler "https://github.com/vishnusharma03/pipeshub-ai/blob/main/backend/python/app/services/scheduler/redis_scheduler/redis_scheduler.py"
    click Scheduler "https://github.com/vishnusharma03/pipeshub-ai/blob/main/backend/python/app/services/scheduler/scheduler_factory.py"
    click ConnectorMain "https://github.com/vishnusharma03/pipeshub-ai/blob/main/backend/python/app/connectors_main.py"
    click IndexMain "https://github.com/vishnusharma03/pipeshub-ai/blob/main/backend/python/app/indexing_main.py"
    click QueryMain "https://github.com/vishnusharma03/pipeshub-ai/blob/main/backend/python/app/query_main.py"
    click Deployment "https://github.com/vishnusharma03/pipeshub-ai/blob/main/deployment/docker-compose/docker-compose.dev.yml"
    click Deployment "https://github.com/vishnusharma03/pipeshub-ai/blob/main/deployment/docker-compose/docker-compose.prod.yml"
    click Deployment "https://github.com/vishnusharma03/pipeshub-ai/blob/main/deployment/docker-compose/docker-compose.ollama-dev.yml"
    click NodeDocker "https://github.com/vishnusharma03/pipeshub-ai/tree/main/backend/nodejs/apps/Dockerfile"
    click PythonDocker "https://github.com/vishnusharma03/pipeshub-ai/tree/main/backend/python/Dockerfile"
    click CICD "https://github.com/vishnusharma03/pipeshub-ai/blob/main/.github/workflows/main.yml"

    %% Styles
    classDef service fill:#cceeff,stroke:#007acc,stroke-width:2px
    classDef database fill:#ccffcc,stroke:#2e7d32,stroke-width:2px
    classDef external fill:#ffe0b2,stroke:#e65100,stroke-width:2px
    classDef broker fill:#fff3cd,stroke:#f0ad4e,stroke-width:2px
    classDef observability fill:#e1bee7,stroke:#6a1b9a,stroke-width:2px
```
