How kimhungbui/khoj works
Main Components
• Client Interfaces:
• Multiple “interfaces” directory under src/interface that contains clients for Android, desktop, Emacs, Obsidian, and web.
• Each of these clients has its own tech stack (for example, Android uses Gradle/Java/Kotlin while the web interface uses Next.js with TypeScript/React).
• Backend Server & API Layer:
• The core resides in src/khoj, which follows a modular monolith pattern. It includes typical server files (manage.py, settings.py, asgi.py) and domain directories like “app”, “database”, “routers”, “processor”, “utils”, etc.
• The “routers” directory hints at REST-like API endpoints for chat, agents, subscriptions, authentication, and more.
• Processors & Content Handlers:
• The processor directory contains submodules to convert various document formats (docx, pdf, markdown, plaintext, etc.) into a standard “entry” format.
• There are dedicated modules for online and offline conversation processing (including integration with providers such as OpenAI, Anthropic, Google, etc.).
• Database:
• The database directory (with migrations, models, management commands) represents the data layer where user data, conversation logs, configurations, and more are persisted.
• Telemetry & Analytics:
• A separate component exists in src/telemetry to capture usage metrics or diagnostics, likely playing a role in monitoring or analytics.
• External Integrations:
• Additional integrations include email interfaces (located within src/khoj/interface/email), search filters, and tool integrations (e.g., online search, code execution, voice chat).
• The project also has CI/CD pipelines and Docker-related files (Dockerfile, docker-compose.yml) to facilitate deployment and containerization.
Relationships & Data Flows
• Client Interfaces ↔ API Gateway:
• Backend Service Components:
• Within the backend, illustrate a modular division:
• “Routers” for exposing APIs to client interfaces.
• “Database” for data persistence.
• “Utilities/Helpers” supporting common application logic.
• Telemetry:
• External Services:
Architectural Patterns & Key Design Principles
• Modular Monolith:
• The backend is organized into modules (routers, processors, database, utils) that communicate internally even though they operate as one unified system.
• Multi-Interface Client Architecture:
• Separation of multiple client apps, each built with its own technology stack, yet unified through common API endpoints.
• Containerization & Deployment:
• CI/CD Pipeline & Automated Testing:
• While not part of runtime architecture, the existence of extensive GitHub workflows (CI, Docker builds, tests, etc.) can be noted for how the project ensures continuous integration and reliability.
Technologies, Frameworks & Libraries
• Backend: Python (likely Django given the presence of manage.py, settings.py, migrations), using ASGI for scaling.
• Frontend:
• Android interfaces using Gradle & Java/Kotlin,
• Desktop interfaces likely built with Electron-like stacks,
• Web interfaces using Next.js with TypeScript/React.
• Deployment and Containerization: Docker, docker-compose, and GitHub Actions for CI/CD.
• Content Processing: Specialized Python scripts/modules for processing docx, PDF, markdown, etc.