# timescale/pgai architecture

> Architecture diagram of the GitHub repository timescale/pgai, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-06). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/timescale/pgai
- Repository: https://github.com/timescale/pgai

## Overview

pgai turns PostgreSQL into a retrieval engine: applications configure vectorizers, workers process queued source data into embeddings and write results back for search and RAG. Its Semantic Catalog adds schema description, semantic search, and natural-language-to-SQL capabilities. The README documents the worker lifecycle; source-index references and excerpts support the shown internal calls. Several implementation details, including unsampled worker and catalog modules, are represented only at subsystem level.

Main entry points: [`projects/pgai/pgai/semantic_catalog/semantic_catalog.py`](https://github.com/timescale/pgai/blob/main/projects/pgai/pgai/semantic_catalog/semantic_catalog.py) (Catalog Management), [`projects/pgai/pgai/vectorizer/worker.py`](https://github.com/timescale/pgai/blob/main/projects/pgai/pgai/vectorizer/worker.py) (Vectorizer Worker), [`projects/pgai/pgai/semantic_catalog/search.py`](https://github.com/timescale/pgai/blob/main/projects/pgai/pgai/semantic_catalog/search.py) (Semantic Search), [`projects/pgai/pgai/cli.py`](https://github.com/timescale/pgai/blob/main/projects/pgai/pgai/cli.py) (Command Line)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_vector["Vectorization"]
  node_vector_config["Vectorizer Configuration"]
  node_worker["Vectorizer Worker<br/>[worker.py]"]
  node_pipeline["Vector Processing<br/>[vectorizer.py]"]
  node_chunking["Chunking<br/>[chunking.py]"]
  node_embedding["Embedding Batches<br/>[embeddings.py]"]
  node_provider_adapters["Provider Adapters<br/>[litellm.py]"]
  node_tracking["Worker Tracking<br/>[worker_tracking.py]"]
end

subgraph group_catalog["Semantic Catalog"]
  node_catalog["Catalog Management"]
  node_describe["Schema Description<br/>[describe.py]"]
  node_catalog_file["Catalog Import Export<br/>[file.py]"]
  node_catalog_vectorizer["Catalog Embeddings<br/>[vectorizer.py]"]
  node_catalog_search["Semantic Search<br/>[search.py]"]
  node_sql_generation["Text-to-SQL<br/>[gen_sql.py]"]
  node_catalog_models["Catalog Models<br/>[models.py]"]
end

subgraph group_runtime["Runtime and Setup"]
  node_cli["Command Line<br/>[cli.py]"]
  node_installer["Database Installer<br/>[install.py]"]
end

node_app(("Application"))
node_postgres[("PostgreSQL")]
node_embedding_api["Embedding Providers"]

node_app -->|"configures"| node_postgres
node_app -->|"defines"| node_vector_config
node_vector_config -->|"stores configuration"| node_postgres
node_worker -->|"reads queue"| node_postgres
node_worker -->|"processes work"| node_pipeline
node_pipeline -->|"chunks text"| node_chunking
node_pipeline -->|"requests embeddings"| node_embedding
node_embedding -.->|"dispatches batches"| node_provider_adapters
node_provider_adapters -.->|"calls provider"| node_embedding_api
node_worker -->|"writes results"| node_postgres
node_worker -->|"reports progress"| node_tracking
node_app -->|"queries vectors"| node_postgres
node_cli -.->|"runs worker"| node_worker
node_app -->|"installs database components"| node_installer
node_cli -->|"installs components"| node_installer
node_cli -->|"invokes catalog operations"| node_catalog
node_catalog -.->|"describes schema"| node_describe
node_describe -->|"uses catalog files"| node_catalog_file
node_catalog -->|"vectorizes catalog"| node_catalog_vectorizer
node_catalog -->|"searches catalog"| node_catalog_search
node_catalog_search -->|"queries embeddings"| node_postgres
node_catalog -->|"generates SQL"| node_sql_generation
node_sql_generation -->|"retrieves context"| node_catalog_search
node_sql_generation -->|"validates SQL"| node_postgres
node_catalog -->|"uses models"| node_catalog_models
node_catalog_search -->|"returns records"| node_catalog_models

click node_vector_config "https://github.com/timescale/pgai/blob/main/projects/pgai/pgai/vectorizer/create_vectorizer.py"
click node_worker "https://github.com/timescale/pgai/blob/main/projects/pgai/pgai/vectorizer/worker.py"
click node_pipeline "https://github.com/timescale/pgai/blob/main/projects/pgai/pgai/vectorizer/vectorizer.py"
click node_chunking "https://github.com/timescale/pgai/blob/main/projects/pgai/pgai/vectorizer/chunking.py"
click node_embedding "https://github.com/timescale/pgai/blob/main/projects/pgai/pgai/vectorizer/embeddings.py"
click node_provider_adapters "https://github.com/timescale/pgai/blob/main/projects/pgai/pgai/vectorizer/embedders/litellm.py"
click node_tracking "https://github.com/timescale/pgai/blob/main/projects/pgai/pgai/vectorizer/worker_tracking/worker_tracking.py"
click node_cli "https://github.com/timescale/pgai/blob/main/projects/pgai/pgai/cli.py"
click node_installer "https://github.com/timescale/pgai/blob/main/projects/pgai/pgai/_install/install.py"
click node_catalog "https://github.com/timescale/pgai/blob/main/projects/pgai/pgai/semantic_catalog/semantic_catalog.py"
click node_describe "https://github.com/timescale/pgai/blob/main/projects/pgai/pgai/semantic_catalog/describe.py"
click node_catalog_file "https://github.com/timescale/pgai/blob/main/projects/pgai/pgai/semantic_catalog/file.py"
click node_catalog_vectorizer "https://github.com/timescale/pgai/blob/main/projects/pgai/pgai/semantic_catalog/vectorizer/vectorizer.py"
click node_catalog_search "https://github.com/timescale/pgai/blob/main/projects/pgai/pgai/semantic_catalog/search.py"
click node_sql_generation "https://github.com/timescale/pgai/blob/main/projects/pgai/pgai/semantic_catalog/gen_sql.py"
click node_catalog_models "https://github.com/timescale/pgai/blob/main/projects/pgai/pgai/semantic_catalog/models.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_vector_config,node_worker,node_pipeline,node_chunking,node_embedding,node_provider_adapters,node_tracking toneBlue
class node_catalog,node_describe,node_catalog_file,node_catalog_vectorizer,node_catalog_search,node_sql_generation,node_catalog_models,node_postgres toneAmber
class node_cli,node_installer toneMint
class node_app,node_embedding_api toneIndigo
```

## Components

### Vectorization

- [Vectorizer Configuration](https://github.com/timescale/pgai/blob/main/projects/pgai/pgai/vectorizer/create_vectorizer.py): `projects/pgai/pgai/vectorizer/create_vectorizer.py`
- [Vectorizer Worker](https://github.com/timescale/pgai/blob/main/projects/pgai/pgai/vectorizer/worker.py): `projects/pgai/pgai/vectorizer/worker.py`
- [Vector Processing](https://github.com/timescale/pgai/blob/main/projects/pgai/pgai/vectorizer/vectorizer.py): `projects/pgai/pgai/vectorizer/vectorizer.py`
- [Chunking](https://github.com/timescale/pgai/blob/main/projects/pgai/pgai/vectorizer/chunking.py): `projects/pgai/pgai/vectorizer/chunking.py`
- [Embedding Batches](https://github.com/timescale/pgai/blob/main/projects/pgai/pgai/vectorizer/embeddings.py): `projects/pgai/pgai/vectorizer/embeddings.py`
- [Provider Adapters](https://github.com/timescale/pgai/blob/main/projects/pgai/pgai/vectorizer/embedders/litellm.py): `projects/pgai/pgai/vectorizer/embedders/litellm.py`
- [Worker Tracking](https://github.com/timescale/pgai/blob/main/projects/pgai/pgai/vectorizer/worker_tracking/worker_tracking.py): `projects/pgai/pgai/vectorizer/worker_tracking/worker_tracking.py`

### Semantic Catalog

- [Catalog Management](https://github.com/timescale/pgai/blob/main/projects/pgai/pgai/semantic_catalog/semantic_catalog.py): `projects/pgai/pgai/semantic_catalog/semantic_catalog.py`
- [Schema Description](https://github.com/timescale/pgai/blob/main/projects/pgai/pgai/semantic_catalog/describe.py): `projects/pgai/pgai/semantic_catalog/describe.py`
- [Catalog Import Export](https://github.com/timescale/pgai/blob/main/projects/pgai/pgai/semantic_catalog/file.py): `projects/pgai/pgai/semantic_catalog/file.py`
- [Catalog Embeddings](https://github.com/timescale/pgai/blob/main/projects/pgai/pgai/semantic_catalog/vectorizer/vectorizer.py): `projects/pgai/pgai/semantic_catalog/vectorizer/vectorizer.py`
- [Semantic Search](https://github.com/timescale/pgai/blob/main/projects/pgai/pgai/semantic_catalog/search.py): `projects/pgai/pgai/semantic_catalog/search.py`
- [Text-to-SQL](https://github.com/timescale/pgai/blob/main/projects/pgai/pgai/semantic_catalog/gen_sql.py): `projects/pgai/pgai/semantic_catalog/gen_sql.py`
- [Catalog Models](https://github.com/timescale/pgai/blob/main/projects/pgai/pgai/semantic_catalog/models.py): `projects/pgai/pgai/semantic_catalog/models.py`

### Runtime and Setup

- [Command Line](https://github.com/timescale/pgai/blob/main/projects/pgai/pgai/cli.py): `projects/pgai/pgai/cli.py`
- [Database Installer](https://github.com/timescale/pgai/blob/main/projects/pgai/pgai/_install/install.py): `projects/pgai/pgai/_install/install.py`

### Other components

- **Application**
- **PostgreSQL**
- **Embedding Providers**

## Connections

- Application → PostgreSQL: configures
- Application → Vectorizer Configuration: defines
- Vectorizer Configuration → PostgreSQL: stores configuration
- Vectorizer Worker → PostgreSQL: reads queue
- Vectorizer Worker → Vector Processing: processes work (evidence: `projects/pgai/pgai/vectorizer/worker.py`)
- Vector Processing → Chunking: chunks text (evidence: `projects/pgai/pgai/vectorizer/vectorizer.py`)
- Vector Processing → Embedding Batches: requests embeddings
- Embedding Batches → Provider Adapters: dispatches batches (evidence: `projects/pgai/pgai/vectorizer/embedders/litellm.py`)
- Provider Adapters → Embedding Providers: calls provider
- Vectorizer Worker → PostgreSQL: writes results
- Vectorizer Worker → Worker Tracking: reports progress (evidence: `projects/pgai/pgai/vectorizer/worker.py`)
- Application → PostgreSQL: queries vectors
- Command Line → Vectorizer Worker: runs worker
- Application → Database Installer: installs database components
- Command Line → Database Installer: installs components (evidence: `projects/pgai/pgai/cli.py`)
- Command Line → Catalog Management: invokes catalog operations (evidence: `projects/pgai/pgai/cli.py`)
- Catalog Management → Schema Description: describes schema
- Schema Description → Catalog Import Export: uses catalog files (evidence: `projects/pgai/pgai/semantic_catalog/describe.py`)
- Catalog Management → Catalog Embeddings: vectorizes catalog (evidence: `projects/pgai/pgai/semantic_catalog/semantic_catalog.py`)
- Catalog Management → Semantic Search: searches catalog (evidence: `projects/pgai/pgai/semantic_catalog/semantic_catalog.py`)
- Semantic Search → PostgreSQL: queries embeddings (evidence: `projects/pgai/pgai/semantic_catalog/search.py`)
- Catalog Management → Text-to-SQL: generates SQL (evidence: `projects/pgai/pgai/semantic_catalog/semantic_catalog.py`)
- Text-to-SQL → Semantic Search: retrieves context (evidence: `projects/pgai/pgai/semantic_catalog/gen_sql.py`)
- Text-to-SQL → PostgreSQL: validates SQL (evidence: `projects/pgai/pgai/semantic_catalog/gen_sql.py`)
- Catalog Management → Catalog Models: uses models (evidence: `projects/pgai/pgai/semantic_catalog/semantic_catalog.py`)
- Semantic Search → Catalog Models: returns records (evidence: `projects/pgai/pgai/semantic_catalog/search.py`)
