How csharikrishna/knowledgedb works
1. Repo shape
- This is a monorepo centered on a Node.js backend plus multiple React clients and SDKs.
- The core product is KnowledgeDB: a self-hosted, file-backed NoSQL DBaaS with graph, search, memory, webhook, and analytics features.
2. Backend entry points
- server/server.js is the HTTP server bootstrap.
- server/app.js holds the Express application wiring.
- The backend is organized by concern: routes define endpoints, controllers handle request logic, middleware enforces security/validation, and utils contain the actual engines.
3. Primary request path
- External clients hit Express routes under /auth, /db, /search, /graph, /memory, /triggers, /webhooks, /vectors, /analytics, and admin/public endpoints.
- Routes delegate to controllers, which coordinate auth, data access, and higher-level behaviors like graph traversal or hybrid search.
- Utility modules implement the domain logic; this keeps HTTP concerns separate from storage/search/graph mechanics.
4. Persistence model
- The system uses pure JSON file storage rather than an external database, matching the “zero external dependencies” design in the README.
- Supporting utilities like fileHandler, documentHelper, indexManager, and queryEngine imply a file-based document store with indexing and query support.
- A data/ directory is mentioned in the README as auto-created storage.
5. Authentication and tenancy boundary
- authController, jwtHelper, and jwtMiddleware implement user auth with JWTs.
- apiKeyMiddleware and apiKeyGenerator support scoped API keys for programmatic access.
- adminMiddleware separates system administration from normal tenant usage.
- This creates a clear boundary between end-user database operations and privileged admin/system operations.
6. Document CRUD and database management
- crudController and databaseController cover database creation plus collection/document CRUD.
- The public data model is database → collection → document, with support for version history and query-by-collection.
- This is the base layer that everything else builds on: graph extraction, search indexing, memory, analytics, triggers, and webhooks all sit downstream of document writes.
7. Knowledge graph, hybrid search, and vector-style features
- graphEngine and graphController provide traversal, path, search, and stats over the auto-built knowledge graph.
- bm25Engine, hybridSearch, and searchController indicate keyword search fused with graph relevance scoring.
- vectorEngine and vectors routes suggest embedding/vector-related functionality alongside the main search stack.
- enhancedSearchController likely combines these into richer retrieval flows such as GraphRAG.
8. AI memory and conversational retrieval
- memoryController and memoryEngine implement session-scoped memory storage and recall.
- The architecture treats memory as a first-class backend subsystem, not just a client-side cache.
- In practice, memory likely feeds GraphRAG or search relevance when generating context-aware responses.
9. Realtime, triggers, and outbound delivery
- sseManager provides Server-Sent Events for live updates from the backend to clients.
- triggerEngine and triggerController manage conditional event rules inside the system.
- webhookDispatcher and webhookController push signed webhook deliveries outward, so the backend can notify external systems when data changes.
10. Analytics and graph-oriented reporting
- analyticsEngine and analyticsController provide aggregation-style reporting such as groupBy, sum, avg, min/max, and time series.
- This is a separate read/compute path from CRUD and search, but it operates on the same stored documents.
- analytics is exposed as an API capability rather than a background batch job.
11. Frontend surfaces
- web-explorer/ is the primary product UI in the README: a React app with welcome, login, dashboard, graph explorer, search, memory browser, GraphRAG tester, and admin panel pages.
- admin-dashboard/ and user-dashboard/ are additional React apps present in the repo and appear to be legacy or alternative dashboards.
- These UIs talk to the backend through src/services/api.js clients and are separated by role and feature area.
12. SDK and programmatic access
- sdk/knowledgedb.js and sdk/knowledgedb.py expose the backend as a consumable client library for Node.js and Python.
- This makes the backend usable both interactively through the web UI and directly from application code.
13. Operational and delivery concerns
- Dockerfile, docker-compose.yml, and render.yaml show the project is packaged for containerized and platform deployment.
- scripts/seed.js, scripts/seed-advanced.js, and scripts/integration-test.js support demo data and verification.
- The surrounding docs focus heavily on deployment, readiness, and integration, indicating the repo is intended to be run as a self-hosted product rather than a library only.
14. Overall architecture diagram takeaway
- The key diagram should show: clients/SDKs → Express API layer → controllers → core engines/utils → JSON file storage, with side paths for graph indexing, search/memory/analytics, and outbound SSE/webhooks.
- Admin functions and user functions share the same backend but are separated by auth, API keys, and middleware-based authorization.
Main entry points: app.js Express App · fileHandler.js File Storage · crudController.js Data Ctrl · search.js Search & Graph
Components 30
Clients
Human-facing apps and programmatic SDKs
- Web Explorer · Primary product UI with dashboard, search, graph, memory, and GraphRAG surfaces
web-explorer/src/App.jsx - Admin Dashboard · Admin-focused React console for system and data oversight
admin-dashboard/src/App.js - User Dashboard · User account and database management UI
user-dashboard/src/App.js - Node SDK · JavaScript client library for programmatic access
sdk/knowledgedb.js - Python SDK · Python client library for programmatic access
sdk/knowledgedb.py
Backend
Node.js API, authorization, and domain engines
- Server Boot · HTTP server entrypoint
server/server.js - Express App · Express application setup and middleware wiring
server/app.js - Auth API · Login, signup, JWT flows, and session boundaries
server/routes/auth.js - Data API · Database, CRUD, and public data endpoints
server/routes/database.js - Search & Graph · Search, graph, enhanced retrieval, and vector endpoints
server/routes/search.js - Memory API · Session memory endpoints
server/routes/memory.js - Events API · Triggers and webhook endpoints
server/routes/triggers.js - Analytics API · Analytic query endpoints
server/routes/analytics.js - Admin API · Privileged system administration endpoints
server/routes/admin.js - Auth Ctrl · Authentication and account request handling
server/controllers/authController.js - Data Ctrl · Database and document CRUD orchestration
server/controllers/crudController.js - Search Ctrl · Keyword, hybrid, and enhanced search orchestration
server/controllers/searchController.js - Graph Ctrl · Graph traversal, path, and stats requests
server/controllers/graphController.js - Memory Ctrl · Session memory recall and persistence requests
server/controllers/memoryController.js - Analytics Ctrl · Aggregation and reporting requests
server/controllers/analyticsController.js - Admin Ctrl · Privileged admin operations
server/controllers/adminController.js - Search Engine · BM25, hybrid ranking, and query evaluation
server/utils/hybridSearch.js - Graph Engine · Knowledge graph construction and traversal
server/utils/graphEngine.js - Memory Engine · Conversation and session memory handling
server/utils/memoryEngine.js - Analytics Engine · GroupBy and metric computations over stored data
server/utils/analyticsEngine.js - Triggers & Webhooks · Trigger evaluation and outbound webhook delivery
server/utils/webhookDispatcher.js
Storage
File-backed persistence and indexing
- File Storage · File-backed document persistence and helpers
server/utils/fileHandler.js - Indexing Core · Document helpers, indexes, and query execution
server/utils/indexManager.js
Ops
Packaging, seeding, and verification
- Deploy Pack · Container and platform deployment definitions
Dockerfile - Seed Tools · Demo data loading and integration verification
scripts/seed.js
Connections 37
- Web Explorer → Express App: API calls · UI clients call the backend API
- Admin Dashboard → Express App: API calls · Admin console uses the same backend
- User Dashboard → Express App: API calls · User console uses the same backend
- Node SDK → Express App: programmatic access · Node SDK wraps backend endpoints
- Python SDK → Express App: programmatic access · Python SDK wraps backend endpoints
- Server Boot → Express App: starts · HTTP server boots the Express app
- Express App → Auth API: mounts · Route wiring for auth flows
- Express App → Data API: mounts · Route wiring for database and CRUD
- Express App → Search & Graph: mounts · Route wiring for search, graph, and vectors
- Express App → Memory API: mounts · Route wiring for memory endpoints
- Express App → Events API: mounts · Route wiring for triggers and webhooks
- Express App → Analytics API: mounts · Route wiring for analytics endpoints
- Express App → Admin API: mounts · Route wiring for admin endpoints
- Auth API → Auth Ctrl: delegates · Auth routes delegate request logic
- Data API → Data Ctrl: delegates · CRUD routes delegate request logic
- Search & Graph → Search Ctrl: delegates · Search routes delegate request logic
- Search & Graph → Graph Ctrl: delegates · Graph routes delegate request logic
- Memory API → Memory Ctrl: delegates · Memory routes delegate request logic
- Events API → Admin Ctrl: delegates · Event-related administrative flows and dispatch coordination
- Analytics API → Analytics Ctrl: delegates · Analytics routes delegate request logic
- Admin API → Admin Ctrl: delegates · Admin routes delegate request logic
- Auth Ctrl → File Storage: auth state · Users and tokens are persisted in file-backed storage
- Data Ctrl → File Storage: reads/writes · Document CRUD uses JSON file persistence
- Data Ctrl → Indexing Core: updates · Writes refresh indexes and query structures
- Search Ctrl → Search Engine: uses · Search controller runs retrieval and ranking
- Search Engine → Indexing Core: queries · Search uses indexes and query execution
- Graph Ctrl → Graph Engine: uses · Graph controller runs traversal and stats
- Graph Engine → File Storage: derives from · Graph is built from stored documents
- Memory Ctrl → Memory Engine: uses · Memory controller stores and recalls context
- Memory Engine → File Storage: persists · Memory is stored in file-backed data
- Analytics Ctrl → Analytics Engine: uses · Analytics controller computes aggregates
- Analytics Engine → File Storage: reads · Analytics operates on stored documents
- Admin Ctrl → File Storage: admin reads/writes · Admin operations inspect and manage stored data
- Data Ctrl → Triggers & Webhooks: emits · Writes can trigger downstream events and webhooks
- Triggers & Webhooks → File Storage: tracks state · Trigger and delivery metadata is backed by files
- Deploy Pack → Server Boot: packages · Containerization packages the backend server
- Seed Tools → File Storage: initializes · Seed scripts populate the file-backed store