How tjmlabs/colivara works
Main Components
• API Server (Django-based):
• Located under the “web” directory.
• Contains two main submodules: “accounts” (e.g., user management, authentication) and “api” (document and collection management, search endpoints).
• Database:
• PostgreSQL with the pgVector extension for embedding storage.
• External Embeddings Service:
• Runs separately (ColiVarE repo) and provides GPU-powered generation of visual embeddings.
• Client SDKs:
• Python and TypeScript SDKs (colivara-py and colivara-ts) that interact with the REST API.
• Docker & CI/CD Infrastructure:
• Docker Compose files (“docker-compose.yml” and “docker-compose-prod.yml”) indicate container orchestration.
• GitHub workflows in the “.github/workflows” directory for testing, deployment, and release processes.
• Additional Supporting Elements:
• Documentation (in docs folder and external URLs).
• Configuration files (environment variables in .env.dev, example-nginx.conf).
Relationships and Interactions
• External Clients (via SDKs or direct REST calls) invoke endpoints on the API Server.
• The API Server processes requests such as document indexing and search.
• When a document is uploaded or indexed, the API Server forwards the file (or URL, base64 content) to the Embeddings Service, which generates embeddings.
• The API Server then stores and retrieves document information and embeddings from the Postgres database with pgVector.
• User management and authentication logic is handled within the “accounts” module.
• CI/CD pipelines (via GitHub workflows) manage automated testing, deployment, and release flows for the codebase.
Architectural Patterns and Design Principles
• Layered Architecture: Separate concerns between API logic (views, controllers), business logic (models, migrations, management commands), and data persistence (Postgres).
• Microservices/Modular Integration: The external Embeddings Service serves as a loosely coupled microservice that can be swapped or updated independently.
• Containerization: Docker Compose is used to orchestrate the API server, database, and (in local development) possibly the Embeddings Service.
• Convention Over Configuration: As noted in the README, the API design uses opinionated defaults to ease development.
• Extensibility: The SDKs and REST API design permit clients to choose between using the provided SDKs or directly hitting the endpoints, which is crucial for integration flexibility.
Technologies
• Django framework (Python-based web framework) for the REST API.
• PostgreSQL with pgVector for vector storage.
• Docker for container orchestration.
• CI/CD workflows powered by GitHub Actions.
• External GPU-powered Embeddings Service using vision models.
• Python and TypeScript SDKs for client integration.