How vishnusharma03/pipeshub-ai works
Project Type and Purpose
• This is a full-stack, modular, microservices-style Workplace AI platform.
• Frontend: React + TypeScript (Vite), renders UI for search, configuration, dashboards.
• Backend Node.js: REST API gateway, authentication, configuration management, crawling manager, enterprise search, notification, storage, tokens, user management modules.
• Backend Python: data ingestion, connectors (Google, Atlassian, Microsoft, S3, local KB, etc.), indexing pipeline, search services, AI agents, Celery tasks, messaging consumers/producers.
• Infrastructure: Docker Compose deployment, message broker (Kafka), databases (ArangoDB, MongoDB, Redis), vector DB (Qdrant), key-value store (etcd), Prometheus metrics.
File Structure Analysis
• Top-level:
• frontend/ (React UI)
• backend/nodejs/apps (gateway API service)
• backend/python/app (ingestion, connectors, indexing, search)
• deployment/docker-compose (dev/prod compose files)
• Patterns:
• Modular code per domain (auth, storage, crawling, enterprise_search, knowledge_base, etc.).
• Layered structure: controllers → services → repositories/providers → utils.
• Hexagonal/ports-and-adapters: keyValueStoreFactory, messaging factory, graph_db_factory, vector_db_factory.
• Messaging: Kafka consumers/producers for asynchronous events.
• Scheduler: Celery + Redis scheduler.
• CI/CD: GitHub workflows, Dockerfiles.
Key README Insights
• Frontend ↔ API Gateway (Node.js)
• Connector Services (Python) → Message Broker → Indexer/Searcher
• Datastores: Knowledge Graph (ArangoDB), Document DB (MongoDB), KV Store (etcd/Redis), Vector DB (Qdrant)
• Observability: Prometheus
• External Services/APIs: Google, Microsoft, Slack, Jira, Confluence, etc.
• Deployment via Docker Compose for cloud or on-prem.
• Frontend (React UI)
• API Gateway (Node.js service)
• Connector Workers (Python processes/agents)
• Asynchronous Message Broker (Kafka)
• Indexing & Search Engine (Python indexing service, Celery tasks)
• Datastores:
– Knowledge Graph DB (ArangoDB)
– Document Store (MongoDB)
– Key-Value Store (etcd/Redis)
– Vector DB (Qdrant)
• External Connectors (Google Workspace, Microsoft 365, Slack, Jira, Confluence, S3, local KB)
• Observability (Prometheus)
1. User browser → Frontend → API Gateway (HTTP/REST, JWT/SAML auth)
2. API Gateway ↔ Datastores (ArangoDB, MongoDB, etc.) for runtime queries and configuration
3. API Gateway → publishes events → Kafka topics (e.g., connect, sync, token)
4. Connector Workers subscribe to Kafka, pull data from External Services, transform and push to Indexer
5. Indexer (Celery tasks) writes processed documents/entities to ArangoDB, MongoDB, Qdrant
6. Search API queries ArangoDB and Vector DB, returns ranked results with citations
7. Observability metrics piped to Prometheus
c. Architectural Patterns & Principles:
• Microservices & Modularization: Each domain module is loosely coupled, independently deployable.
• Hexagonal Architecture: Adapters for connectors, stores, messaging.
• Event-Driven: Kafka for decoupling ingestion, sync, and indexing.
• Asynchronous Processing: Celery + Redis for long-running jobs (OCR, PDF parsing, AI model inference).
• No-Code App Builders: modular “builders” in Python for custom workflows.
• Security: OAuth, JWT, SAML, RBAC middlewares.
d. Relevant Technologies & Frameworks:
• Frontend: React, TypeScript, Vite, MUI, Zustand/Context API
• API Gateway: Node.js, TypeScript, Express, InversifyJS (DI container), Prometheus instrumentation, Kafka client, ArangoJS, Mongoose, etc.
• Ingestion/Search: Python, FastAPI, Celery, ai_arango_service, Qdrant, Pymupdf, docling, OpenAI SDK (or Ollama), ArangoDB Python driver
• Messaging: Kafka (brokers, topics, producers/consumers)
• Datastores: ArangoDB (graph), MongoDB (documents), Redis (KV cache, scheduler), etcd (config store), Qdrant (vector embeddings)
• Deployment: Docker, Docker Compose, GitHub Actions CI