# weaviate/verba architecture

> Architecture diagram of the GitHub repository weaviate/verba, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-16). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/weaviate/verba
- Repository: https://github.com/weaviate/verba

## Overview

##### Project Type and Purpose
- This is a full-stack, open-source RAG (Retrieval-Augmented Generation) web application.
- Frontend: Next.js/React (in the `frontend` directory) provides a customizable UI for ingesting data, configuring pipelines, and chatting with documents.
- Backend: Python package `goldenverba` (in the `goldenverba` directory) implements data ingestion, chunking, embedding, retrieval, and generation pipelines and exposes a FastAPI server (`goldenverba/server/api.py`) plus a CLI (`goldenverba/server/cli.py`).
- It integrates a vector database (Weaviate), various chunkers, embedders, generators, and external services (OpenAI, Cohere, AssemblyAI, HuggingFace, Ollama, etc.).

##### High-Level File Structure
- Top-level:
- `frontend/` – Next.js app
- `goldenverba/` – Python library (“components”, “server”, CLI, helpers)
- `tests/` – Python unit tests
- `Dockerfile` & `docker-compose.yml` – container and service definitions
- Config and metadata: `.env.example`, `setup.py`, `README.md`, `FRONTEND.md`, `TECHNICAL.md`
- Patterns: MVC-style separation (UI <-> API <-> services), modular plugin system for chunkers/embedders/generators via interfaces, monolithic deployment orchestrated by Docker Compose.

##### Key README Insights
- Describes supported embedding/generation providers and data ingest pipelines.
- Deployment modes: Local (embedded Weaviate), Docker, Cloud, Custom.
- Environment variables configure external services.

a. Main Components
1. Frontend (Next.js)
- Routes/UI pages (Chat, Document Explorer, Ingestion, Settings)
- HTTP/REST calls → Backend API
2. Backend API (FastAPI)
- Authentication & config loader
- Ingestion Service → Readers → Chunkers → Weaviate
- Query Service → Retriever → Generator → Response
3. Python Core (`goldenverba/components`)
- Chunking modules (Token, Semantic, Recursive, Markdown, etc.)
- Embedding modules (OpenAI, Cohere, Upstage, Weaviate, etc.)
- Generation modules (OpenAI, Anthropic, Gemini, etc.)
- Interfaces & managers to orchestrate pipelines
4. Vector Database (Weaviate)
- Stores embeddings and metadata
5. External Services
- LLM providers (OpenAI, Cohere, Anthropic, Ollama…)
- Unstructured IO, Firecrawl, AssemblyAI, GitHub/GitLab readers
b. Relationships & Interactions
- Frontend → (HTTPS) → Backend API
- API loads config/env, dispatches ingestion or query pipelines
- Ingestion pipeline: Reader → Chunker → Embedding → Weaviate
- Query pipeline: Retriever (Weaviate) → Generation (LLM) → Response to Frontend
- CLI orchestrates service startup (invokes FastAPI server)
- Docker Compose wires up services: `verba` container, optional `weaviate` and `ollama` containers
c. Architectural Patterns & Principles
- Plugin architecture via interfaces for chunkers/embedders/generators
- Clear separation of concerns: UI / API / Business Logic / Data Layer
- Environment-driven configuration for service endpoints and keys
- Asynchronous ingestion & querying where supported
d. Technologies & Frameworks
- Frontend: React, Next.js, Tailwind CSS, WebGL shaders (for vector viewer)
- Backend: Python 3.10+, FastAPI, spaCy, Uvicorn, dependency injection
- Data store: Weaviate (embedded or Docker)
- DevOps: Docker, Docker Compose, GitHub Actions

## Diagram (Mermaid)

```mermaid
graph TB
    %% Presentation Layer
    subgraph "Presentation Layer"
        Browser["Browser"]:::ui
        Frontend["Next.js UI"]:::ui
    end

    User["User"]:::external
    User --> Browser
    Browser --> Frontend
    Frontend -->|HTTP/REST| API

    %% Service Layer
    subgraph "Service Layer"
        API["FastAPI Server"]:::api
        CLI["CLI Entry Point"]:::api
        Config["Env Config Loader"]:::infra

        subgraph "Core Components"
            Managers["Ingestion & Query Managers"]:::component
            Interfaces["Component Interfaces"]:::component
            Readers["Document Readers"]:::component
            Chunkers["Chunking Modules"]:::component
            Embedders["Embedding Modules"]:::component
            WeaveEmb["Weaviate Embedder"]:::component
            Retrievers["Retrieval Modules"]:::component
            Generators["Generation Modules"]:::component
            Types["Core Types"]:::component
            Util["Utilities"]:::component
        end
    end

    Frontend --> API
    API --> Config
    API --> Managers
    CLI --> API

    Managers --> Readers
    Managers --> Chunkers
    Chunkers --> Embedders
    Embedders --> WeaveEmb
    WeaveEmb --> Weaviate

    Managers --> Retrievers
    Retrievers --> Weaviate

    Managers --> Generators
    Generators --> LLM

    API -->|response| Frontend

    %% Data Layer
    subgraph "Data Layer"
        Weaviate["Weaviate DB"]:::db
    end
    Weaviate --> API

    %% External Services
    subgraph "External Services"
        LLM["LLM Providers"]:::external
        OtherExt["Other Services"]:::external
    end
    LLM --> API
    OtherExt --> API

    %% Infrastructure
    subgraph "Infrastructure"
        DockerfileNode["Dockerfile"]:::infra
        DockerCompose["docker-compose.yml"]:::infra
    end
    DockerCompose --> API
    DockerCompose --> Weaviate
    DockerCompose --> CLI

    %% Interfaces and Utilities connectivity
    Interfaces --> Readers
    Interfaces --> Chunkers
    Interfaces --> Embedders
    Interfaces --> Generators
    Types --> Interfaces
    Util --> Managers

    %% Click Events
    click Frontend "https://github.com/weaviate/verba/tree/main/frontend/"
    click API "https://github.com/weaviate/verba/blob/main/goldenverba/server/api.py"
    click CLI "https://github.com/weaviate/verba/blob/main/goldenverba/server/cli.py"
    click Managers "https://github.com/weaviate/verba/blob/main/goldenverba/components/managers.py"
    click Chunkers "https://github.com/weaviate/verba/tree/main/goldenverba/components/chunking/"
    click Embedders "https://github.com/weaviate/verba/tree/main/goldenverba/components/embedding/"
    click WeaveEmb "https://github.com/weaviate/verba/blob/main/goldenverba/components/embedding/WeaviateEmbedder.py"
    click Generators "https://github.com/weaviate/verba/tree/main/goldenverba/components/generation/"
    click Readers "https://github.com/weaviate/verba/tree/main/goldenverba/components/reader/"
    click Retrievers "https://github.com/weaviate/verba/tree/main/goldenverba/components/retriever/"
    click Interfaces "https://github.com/weaviate/verba/blob/main/goldenverba/components/interfaces.py"
    click Types "https://github.com/weaviate/verba/blob/main/goldenverba/components/types.py"
    click Util "https://github.com/weaviate/verba/blob/main/goldenverba/components/util.py"
    click Config "https://github.com/weaviate/verba/blob/main/goldenverba/.env.example"
    click DockerfileNode "https://github.com/weaviate/verba/tree/main/Dockerfile"
    click DockerCompose "https://github.com/weaviate/verba/blob/main/docker-compose.yml"

    %% Styles
    classDef ui fill:#a2d2ff,stroke:#000
    classDef api fill:#caffbf,stroke:#000
    classDef component fill:#ffd6a5,stroke:#000
    classDef db fill:#bdb2ff,stroke:#000
    classDef external fill:#ffadad,stroke:#000
    classDef infra fill:#fdffb6,stroke:#000
```
