# tensorchord/vechord architecture

> Architecture diagram of the GitHub repository tensorchord/vechord, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-31). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/tensorchord/vechord
- Repository: https://github.com/tensorchord/vechord

## Overview

##### Main components
- Library Core (vechord package): This is where the heart of the library’s functionality lives. Components include:
- Table and specification definitions (spec.py)
- Database client and load handling (client.py, load.py)
- Data processing modules (augment.py, chunk.py, extract.py, embedding.py, evaluate.py, rerank.py, model.py, log.py)
- The Registry, which manages table registration, dependency injection, and transactions (registry.py)
- The web service creation utility (service.py) that sets up an auto-generated HTTP API.
- External Dependencies & Integrations:
- PostgreSQL database (as the underlying storage and indexing backend)
- External libraries/services for embedding, web requests (illustrated in the examples with HTTPX) and potentially other vector database tools (RaBitQ, WARP, BM25).
- Examples & Tests:
- Illustrative scripts in the “examples” directory that show different use cases and integrations
- Testing modules in the “tests” directory (ensuring that internal modules behave as expected)

##### Relationships and interactions
- The Registry (from registry.py) ties everything together: it is responsible for registering table definitions, creating indices, and coordinating the modules that convert user inputs (or file content in examples) into structured search data.
- Data flows from user-defined functions (decorated with @inject) where external input (e.g., HTML content fetched via HTTPX) is processed by “extractor” and “embedding” modules, then stored into the PostgreSQL database.
- Embedded components such as “Chunker” (chunk.py) and “Augmenter” (augment.py) preprocess and transform text, feeding results into subsequent modules.
- The auto-generated web service (from service.py) creates a WSGI app to expose search endpoints (vector search, keyword search) based on the functions registered by the user.
- Transaction management is handled by the registry’s run method, ensuring that the entire pipeline (data loading, processing, and storing) is executed atomically.

##### Architectural patterns and design principles
- Modularization and separation of concerns: each Python module handles a distinct aspect (e.g., chunking, embedding, evaluation), allowing for flexible integration or replacement.
- Decorator pattern: used to automatically inject database read/write operations into user-defined functions.
- Transactional integrity: using the pipeline concept to combine multiple operations into a consistent transaction.
- Extensibility: by providing common functions and an auto-generated API, users can extend or customize the behavior.

##### Relevant technologies and frameworks
- Python (with type hinting awareness, supported by the py.typed file)
- PostgreSQL (as the backend datastore, including vector search functionality)
- HTTPX (for external web data fetching)
- WSGI (for deploying the web service)
- CI/CD tooling (GitHub workflows for check, pages, release)
- Build tools (Makefile, envd configuration)

##### Elements
- Clear labels for each component: List each module (e.g., Embedding, Chunking, Extractor) and layer (API, core processing, database).

##### Putting it all together
- From here, trace the flow into the Registry, which then interacts with the several processing modules (embedding, chunking, extraction, etc.) before writing/reading to PostgreSQL.
- Transactional pipeline and how decorated functions (using dependency injection) are part of this workflow.
- Lastly, ensure that you capture the testing and documentation aspects (though secondary) as supporting components for the overall system.

## Diagram (Mermaid)

```mermaid
flowchart TD
    UI["User Interaction"]:::action
    DC["Direct Function Calls (Examples)"]:::action
    UI -->|"direct"| DC
    UI -->|"HTTP"| WS_Alias["HTTP Service"]:::core

    subgraph "Library Core"
        SPEC["Specifications (spec.py)"]:::core
        REG["Registry (registry.py)"]:::core
        DBC["Database Client & Load (client.py & load.py)"]:::core
        WS["HTTP Service (service.py)"]:::core
        subgraph "Data Processing Modules"
            AUG["Augmenter (augment.py)"]:::processor
            CHK["Chunker (chunk.py)"]:::processor
            EXT["Extractor (extract.py)"]:::processor
            EMB["Embedding (embedding.py)"]:::processor
            EVAL["Evaluator (evaluate.py)"]:::processor
            RER["Rerank (rerank.py)"]:::processor
            MOD["Model Handler (model.py)"]:::processor
            LOG["Logger (log.py)"]:::processor
        end
    end

    %% Connect external user interaction to Library Core components
    DC --> REG
    WS_Alias --> WS
    WS --> REG

    %% Data flow within Library Core
    REG -->|"registers"| AUG
    AUG -->|"processes"| CHK
    CHK -->|"extracts"| EXT
    EXT -->|"embeds"| EMB
    EMB -->|"evaluates"| EVAL
    EVAL -->|"reranks"| RER
    RER -->|"handles"| MOD
    MOD -->|"logs"| LOG
    REG -->|"manages"| DBC
    DBC -->|"performs load/store"| DB[(PostgreSQL Database)]:::external
    REG -->|"transaction"| DB

    %% External integration for content fetching
    HTTPX["HTTPX (External)"]:::external
    HTTPX -->|"fetches"| EXT

    %% Examples and Tests sections
    EXAMPLES["Examples (Directory)"]:::tests
    TESTS["Tests (Directory)"]:::tests
    DC --> EXAMPLES
    TESTS -->|"verifies"| REG

    %% Click Events
    click SPEC "https://github.com/tensorchord/vechord/blob/main/vechord/spec.py"
    click REG "https://github.com/tensorchord/vechord/blob/main/vechord/registry.py"
    click DBC "https://github.com/tensorchord/vechord/blob/main/vechord/client.py"
    click DBC "https://github.com/tensorchord/vechord/blob/main/vechord/load.py"
    click AUG "https://github.com/tensorchord/vechord/blob/main/vechord/augment.py"
    click CHK "https://github.com/tensorchord/vechord/blob/main/vechord/chunk.py"
    click EXT "https://github.com/tensorchord/vechord/blob/main/vechord/extract.py"
    click EMB "https://github.com/tensorchord/vechord/blob/main/vechord/embedding.py"
    click EVAL "https://github.com/tensorchord/vechord/blob/main/vechord/evaluate.py"
    click RER "https://github.com/tensorchord/vechord/blob/main/vechord/rerank.py"
    click MOD "https://github.com/tensorchord/vechord/blob/main/vechord/model.py"
    click LOG "https://github.com/tensorchord/vechord/blob/main/vechord/log.py"
    click WS "https://github.com/tensorchord/vechord/blob/main/vechord/service.py"
    click EXAMPLES "https://github.com/tensorchord/vechord/tree/main/examples/"
    click TESTS "https://github.com/tensorchord/vechord/tree/main/tests/"
    click REG "https://github.com/tensorchord/vechord/tree/main/vechord/"

    %% Styles
    classDef core fill:#cce5ff,stroke:#004085,stroke-width:2px;
    classDef processor fill:#d4edda,stroke:#155724,stroke-width:2px;
    classDef external fill:#f8d7da,stroke:#721c24,stroke-width:2px;
    classDef action fill:#fff3cd,stroke:#856404,stroke-width:2px;
    classDef tests fill:#d1ecf1,stroke:#0c5460,stroke-width:2px;

    class UI,DC action
    class SPEC,REG,DBC,WS core
    class AUG,CHK,EXT,EMB,EVAL,RER,MOD,LOG processor
    class HTTPX,DB external
    class EXAMPLES,TESTS tests
```
