How weaviate/verba works
Project Type and Purpose
• This is a full-stack, open-source RAG (Retrieval-Augmented Generation) web application.
• Frontend: Next.js/React (in the frontend directory) provides a customizable UI for ingesting data, configuring pipelines, and chatting with documents.
• Backend: Python package goldenverba (in the goldenverba directory) implements data ingestion, chunking, embedding, retrieval, and generation pipelines and exposes a FastAPI server (goldenverba/server/api.py) plus a CLI (goldenverba/server/cli.py).
• It integrates a vector database (Weaviate), various chunkers, embedders, generators, and external services (OpenAI, Cohere, AssemblyAI, HuggingFace, Ollama, etc.).
High-Level File Structure
• Top-level:
• frontend/ – Next.js app
• goldenverba/ – Python library (“components”, “server”, CLI, helpers)
• tests/ – Python unit tests
• Dockerfile & docker-compose.yml – container and service definitions
• Config and metadata: .env.example, setup.py, README.md, FRONTEND.md, TECHNICAL.md
• Patterns: MVC-style separation (UI <-> API <-> services), modular plugin system for chunkers/embedders/generators via interfaces, monolithic deployment orchestrated by Docker Compose.
Key README Insights
• Describes supported embedding/generation providers and data ingest pipelines.
• Deployment modes: Local (embedded Weaviate), Docker, Cloud, Custom.
• Environment variables configure external services.
a. Main Components
1. Frontend (Next.js)
• Routes/UI pages (Chat, Document Explorer, Ingestion, Settings)
• HTTP/REST calls → Backend API
2. Backend API (FastAPI)
• Authentication & config loader
• Ingestion Service → Readers → Chunkers → Weaviate
• Query Service → Retriever → Generator → Response
3. Python Core (goldenverba/components)
• Chunking modules (Token, Semantic, Recursive, Markdown, etc.)
• Embedding modules (OpenAI, Cohere, Upstage, Weaviate, etc.)
• Generation modules (OpenAI, Anthropic, Gemini, etc.)
• Interfaces & managers to orchestrate pipelines
4. Vector Database (Weaviate)
• Stores embeddings and metadata
5. External Services
• LLM providers (OpenAI, Cohere, Anthropic, Ollama…)
• Unstructured IO, Firecrawl, AssemblyAI, GitHub/GitLab readers
b. Relationships & Interactions
• Frontend → (HTTPS) → Backend API
• API loads config/env, dispatches ingestion or query pipelines
• Ingestion pipeline: Reader → Chunker → Embedding → Weaviate
• Query pipeline: Retriever (Weaviate) → Generation (LLM) → Response to Frontend
• CLI orchestrates service startup (invokes FastAPI server)
• Docker Compose wires up services: verba container, optional weaviate and ollama containers
c. Architectural Patterns & Principles
• Plugin architecture via interfaces for chunkers/embedders/generators
• Clear separation of concerns: UI / API / Business Logic / Data Layer
• Environment-driven configuration for service endpoints and keys
• Asynchronous ingestion & querying where supported
d. Technologies & Frameworks
• Frontend: React, Next.js, Tailwind CSS, WebGL shaders (for vector viewer)
• Backend: Python 3.10+, FastAPI, spaCy, Uvicorn, dependency injection
• Data store: Weaviate (embedded or Docker)
• DevOps: Docker, Docker Compose, GitHub Actions