# rapidaai/voice-ai architecture

> Architecture diagram of the GitHub repository rapidaai/voice-ai, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-06-22). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/rapidaai/voice-ai
- Repository: https://github.com/rapidaai/voice-ai

## Overview

1. This repo is a polyglot monorepo for a voice-AI platform, but the architectural center is the Go backend suite under `api/`, plus a React UI under `ui/`, and a separate Python document-ingestion service under `api/document-api`.

2. The main runtime boundaries are four Go services plus one Python service:
   - `web-api` for product-facing CRUD and account/workspace management
   - `assistant-api` for real-time agent orchestration and voice sessions
   - `endpoint-api` for endpoint definition/invocation workflows
   - `integration-api` for provider normalization and external AI integrations
   - `document-api` for knowledge/document ingestion and indexing
   These are deployed independently, with their own configs, migrations, and routers.

3. `web-api` is the control-plane backend for the UI. It manages organizations, projects, users, notifications, vault/connect resources, and proxy endpoints that fan out to the other APIs. It also owns its own relational schema via migrations and uses shared auth/service clients from `pkg/`.

4. `assistant-api` is the core data-plane service. It handles live voice sessions, assistant configuration, deployments, conversations, messages, telemetry, knowledge attachment, tools, and webhooks. Its architecture is layered: HTTP/gRPC-facing `api/` handlers on top, service/use-case code in `internal/services`, and domain entities in `internal/entity`.

5. Inside `assistant-api`, the most important subsystem is the channel/pipeline layer: `internal/channel/*`, `internal/adapters/*`, and `internal/channel/telephony/*` / `webrtc/*`. This is where audio and session events are turned into a real-time pipeline that can accept input, dispatch it through STT/LLM/tooling, and emit output back to the channel.

6. The assistant voice pipeline depends on several low-level audio components:
   - VAD engines in `internal/vad`
   - noise suppression in `internal/denoiser`
   - end-of-speech detectors in `internal/end_of_speech`
   - audio resampling/normalization in `internal/audio` and `internal/normalizer`
   These are not UI concerns; they are latency-critical runtime components that sit directly on the audio stream.

7. Telephony is a major boundary in `assistant-api`. The repo includes adapters for Twilio, Vonage, Telnyx, Exotel, SIP/Asterisk, and WebRTC, with media/session/registration state machines and outbound/inbound call handling. That means the platform is designed to bridge external phone systems into the same assistant pipeline used for web or debugger sessions.

8. Model/provider integration is abstracted heavily. `api/assistant-api/internal/transformer/*` contains provider-specific STT/TTS adapters for vendors like OpenAI, Deepgram, Google, Azure, AssemblyAI, ElevenLabs, Cartesia, NVIDIA, and others; `api/integration-api/internal/caller/*` contains the normalized chat/embedding/reranking/provider layer for LLM and related APIs. Together, these separate “provider specifics” from the higher-level assistant orchestration.

9. `integration-api` acts as a provider gateway. It exposes chat, embedding, reranking, audit logging, and unified provider endpoints, and internally it contains vendor-specific caller implementations and credential verification paths. This service is the main integration boundary between the platform and external AI vendors.

10. `endpoint-api` is the endpoint management/service-execution backend. It stores endpoint definitions, retry/cache configs, versions, provider models, and logs, then exposes invocation and trace retrieval. In the architecture diagram, this is the service for API-style endpoint execution rather than live voice sessions.

11. `document-api` is a separate Python pipeline for knowledge ingestion and indexing. It includes connectors to storage and source systems, extractors/chunkers/cleaners, vector-store and docstore components, Celery workers, and bridges to other services via generated protobuf clients. It depends on infrastructure like PostgreSQL, Redis, and OpenSearch, and is the service to draw for RAG/document processing rather than online voice execution.

12. The React `ui/` package is the management console. It talks to `web-api` and proxy-backed APIs to manage assistants, endpoints, knowledge bases, integrations, users, and observability views. The UI is organized by domain pages and shared components, with many provider/config forms that mirror the backend’s provider abstractions.

13. Shared infrastructure and libraries live in `pkg/` and are used across services. This includes:
   - auth and scope authenticators
   - generated/protobuf and REST clients
   - configuration and connector factories
   - telemetry/logging helpers
   - storage abstractions
   - plugin/provider registries
   - common domain types and utilities
   This layer is the main cross-service reuse boundary.

14. Persistence and external systems are central to the architecture: PostgreSQL for core relational state, Redis for caching/worker coordination, OpenSearch for document/vector search, and object storage backends like S3/GCS/Azure Blob for files and media assets. The services also integrate with many external SaaS/AI providers for speech, LLM, telephony, email, and telemetry.

15. Operationally, the repo is designed for containerized deployment. Each service has its own Dockerfile and env/config file, the root includes Docker Compose and nginx/API-gateway setup, and CI/CD workflows build images and deploy individual services. This supports both fully managed and self-hosted topologies.

16. For an architecture diagram, the clearest top-level picture is: UI and external clients hit the web/API gateway layer; that layer coordinates with assistant, endpoint, integration, and document services; assistant handles realtime audio/session execution via telephony/WebRTC and model providers; document-api handles ingestion/indexing; shared libs and infrastructure sit underneath all services.

Main entry points: [`api/web-api`](https://github.com/rapidaai/voice-ai/tree/main/api/web-api) (web-api), [`api/assistant-api`](https://github.com/rapidaai/voice-ai/tree/main/api/assistant-api) (assistant-api), [`api/integration-api`](https://github.com/rapidaai/voice-ai/tree/main/api/integration-api) (integration-api), [`api/document-api`](https://github.com/rapidaai/voice-ai/tree/main/api/document-api) (document-api)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_group_api["API services"]
  node_web_api["web-api<br/>control-plane API"]
  node_assistant_api["assistant-api<br/>realtime service"]
  node_endpoint_api["endpoint-api<br/>endpoint service"]
  node_integration_api["integration-api<br/>provider gateway"]
  node_document_api["document-api<br/>ingestion pipeline"]
  node_integration_callers["Callers<br/>vendor implementations"]
  node_document_core["Doc pipeline<br/>RAG pipeline"]
  node_document_connectors["Connectors<br/>source/storage adapters"]
  node_cmd_entrypoints["Cmd entrypoints<br/>service launchers"]
end

subgraph group_group_assistant["Assistant internals"]
  node_assistant_pipeline["Pipeline<br/>channel orchestration"]
  node_assistant_telephony["Telephony<br/>phone adapters"]
  node_assistant_webrtc["WebRTC<br/>web session adapter"]
  node_assistant_audio["Audio core<br/>audio runtime"]
  node_assistant_transformers["Transformers<br/>provider adapters"]
  node_assistant_llm["LLM runtime<br/>agent execution"]
end

subgraph group_group_shared["Shared platform"]
  node_pkg_shared["pkg<br/>shared libraries"]
  node_infrastructure[("Infra<br/>state and externals")]
end

subgraph group_group_ui["Management UI"]
  node_ui["UI<br/>React console"]
end

node_ui -->|"manage"| node_web_api
node_web_api -->|"proxy"| node_assistant_api
node_web_api -->|"proxy"| node_endpoint_api
node_web_api -->|"proxy"| node_integration_api
node_web_api -->|"proxy"| node_document_api
node_assistant_api -->|"orchestrates"| node_assistant_pipeline
node_assistant_api -->|"bridges"| node_assistant_telephony
node_assistant_api -->|"streams"| node_assistant_webrtc
node_assistant_pipeline -->|"processes"| node_assistant_audio
node_assistant_pipeline -->|"uses"| node_assistant_transformers
node_assistant_pipeline -->|"executes"| node_assistant_llm
node_assistant_llm -->|"calls"| node_integration_api
node_integration_api -->|"implements"| node_integration_callers
node_document_api -->|"runs"| node_document_core
node_document_api -->|"ingests"| node_document_connectors
node_document_api -->|"calls"| node_integration_api
node_web_api -->|"shares"| node_pkg_shared
node_assistant_api -->|"shares"| node_pkg_shared
node_endpoint_api -->|"shares"| node_pkg_shared
node_integration_api -->|"shares"| node_pkg_shared
node_document_api -->|"shares"| node_pkg_shared
node_cmd_entrypoints -->|"starts"| node_web_api
node_cmd_entrypoints -->|"starts"| node_assistant_api
node_cmd_entrypoints -->|"starts"| node_endpoint_api
node_cmd_entrypoints -->|"starts"| node_integration_api
node_document_api -->|"depends on"| node_infrastructure
node_assistant_api -->|"depends on"| node_infrastructure
node_web_api -->|"depends on"| node_infrastructure
node_endpoint_api -->|"depends on"| node_infrastructure
node_integration_api -->|"depends on"| node_infrastructure

click node_ui "https://github.com/rapidaai/voice-ai/tree/main/ui"
click node_web_api "https://github.com/rapidaai/voice-ai/tree/main/api/web-api"
click node_assistant_api "https://github.com/rapidaai/voice-ai/tree/main/api/assistant-api"
click node_endpoint_api "https://github.com/rapidaai/voice-ai/tree/main/api/endpoint-api"
click node_integration_api "https://github.com/rapidaai/voice-ai/tree/main/api/integration-api"
click node_document_api "https://github.com/rapidaai/voice-ai/tree/main/api/document-api"
click node_assistant_pipeline "https://github.com/rapidaai/voice-ai/tree/main/api/assistant-api/internal/channel/pipeline"
click node_assistant_telephony "https://github.com/rapidaai/voice-ai/tree/main/api/assistant-api/internal/channel/telephony"
click node_assistant_webrtc "https://github.com/rapidaai/voice-ai/tree/main/api/assistant-api/internal/channel/webrtc"
click node_assistant_audio "https://github.com/rapidaai/voice-ai/tree/main/api/assistant-api/internal/audio"
click node_assistant_transformers "https://github.com/rapidaai/voice-ai/tree/main/api/assistant-api/internal/transformer"
click node_assistant_llm "https://github.com/rapidaai/voice-ai/tree/main/api/assistant-api/internal/llm"
click node_integration_callers "https://github.com/rapidaai/voice-ai/tree/main/api/integration-api/internal/caller"
click node_document_core "https://github.com/rapidaai/voice-ai/tree/main/api/document-api/app/core"
click node_document_connectors "https://github.com/rapidaai/voice-ai/tree/main/api/document-api/app/connectors"
click node_pkg_shared "https://github.com/rapidaai/voice-ai/tree/main/pkg"
click node_cmd_entrypoints "https://github.com/rapidaai/voice-ai/tree/main/cmd"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_web_api,node_assistant_api,node_endpoint_api,node_integration_api,node_document_api,node_integration_callers,node_document_core,node_document_connectors,node_cmd_entrypoints toneBlue
class node_assistant_pipeline,node_assistant_telephony,node_assistant_webrtc,node_assistant_audio,node_assistant_transformers,node_assistant_llm toneAmber
class node_pkg_shared,node_infrastructure toneMint
class node_ui toneRose
```

## Components

### API services

Independently deployed backend services under api/.

- [web-api](https://github.com/rapidaai/voice-ai/tree/main/api/web-api): control-plane API · Product-facing backend for organizations, projects, users, notifications, vault/connect resources, and proxy fan-out. · `api/web-api`
- [assistant-api](https://github.com/rapidaai/voice-ai/tree/main/api/assistant-api): realtime service · Core voice assistant backend for live sessions, deployments, conversations, telemetry, tools, knowledge, and webhooks. · `api/assistant-api`
- [endpoint-api](https://github.com/rapidaai/voice-ai/tree/main/api/endpoint-api): endpoint service · Endpoint definition, versioning, retry/cache config, invocation, and trace/log retrieval backend. · `api/endpoint-api`
- [integration-api](https://github.com/rapidaai/voice-ai/tree/main/api/integration-api): provider gateway · Unified external AI provider gateway for chat, embeddings, reranking, and audit logging. · `api/integration-api`
- [document-api](https://github.com/rapidaai/voice-ai/tree/main/api/document-api): ingestion pipeline · Python service for document ingestion, extraction, chunking, indexing, and vector/docstore writes. · `api/document-api`
- [Callers](https://github.com/rapidaai/voice-ai/tree/main/api/integration-api/internal/caller): vendor implementations · Vendor-specific normalized chat, embedding, reranking, and credential verification implementations. · `api/integration-api/internal/caller`
- [Doc pipeline](https://github.com/rapidaai/voice-ai/tree/main/api/document-api/app/core): RAG pipeline · Extraction, cleaning, chunking, splitting, indexing, and docstore/vector-store processing stages. · `api/document-api/app/core`
- [Connectors](https://github.com/rapidaai/voice-ai/tree/main/api/document-api/app/connectors): source/storage adapters · Storage, source-system, and bridge adapters used by the ingestion service. · `api/document-api/app/connectors`
- [Cmd entrypoints](https://github.com/rapidaai/voice-ai/tree/main/cmd): service launchers · Top-level Go service binaries for web, assistant, endpoint, and integration processes. · `cmd`

### Assistant internals

Core realtime voice runtime inside assistant-api.

- [Pipeline](https://github.com/rapidaai/voice-ai/tree/main/api/assistant-api/internal/channel/pipeline): channel orchestration · Realtime session pipeline that routes audio and events between input, STT/LLM/tools, and output. · `api/assistant-api/internal/channel/pipeline`
- [Telephony](https://github.com/rapidaai/voice-ai/tree/main/api/assistant-api/internal/channel/telephony): phone adapters · Inbound/outbound call adapters and state machines for Twilio, Vonage, Telnyx, Exotel, SIP/Asterisk, and related media handling. · `api/assistant-api/internal/channel/telephony`
- [WebRTC](https://github.com/rapidaai/voice-ai/tree/main/api/assistant-api/internal/channel/webrtc): web session adapter · WebRTC streaming and session loops for browser-based assistant sessions. · `api/assistant-api/internal/channel/webrtc`
- [Audio core](https://github.com/rapidaai/voice-ai/tree/main/api/assistant-api/internal/audio): audio runtime · Latency-critical audio processing: VAD, denoising, end-of-speech, resampling, normalization, and buffering. · `api/assistant-api/internal/audio`
- [Transformers](https://github.com/rapidaai/voice-ai/tree/main/api/assistant-api/internal/transformer): provider adapters · Provider-specific STT/TTS implementations and normalization for speech vendors. · `api/assistant-api/internal/transformer`
- [LLM runtime](https://github.com/rapidaai/voice-ai/tree/main/api/assistant-api/internal/llm): agent execution · AgentKit/model execution layer, websocket chat runtime, embeddings, reranking, variables, and tool dispatch. · `api/assistant-api/internal/llm`

### Shared platform

Cross-service packages and infrastructure under pkg/.

- [pkg](https://github.com/rapidaai/voice-ai/tree/main/pkg): shared libraries · Cross-service shared auth, clients, configs, connectors, telemetry, storages, plugins, types, and utilities. · `pkg`
- **Infra**: state and externals · PostgreSQL, Redis, OpenSearch, object storage, and external SaaS/AI providers used across services.

### Management UI

React console under ui/.

- [UI](https://github.com/rapidaai/voice-ai/tree/main/ui): React console · Management console for assistants, endpoints, knowledge, integrations, users, and observability. · `ui`

## Connections

- UI → web-api: manage · UI drives product workflows through the control-plane backend.
- web-api → assistant-api: proxy · Control plane proxies assistant management and related workflows.
- web-api → endpoint-api: proxy · Control plane proxies endpoint management and invocation workflows.
- web-api → integration-api: proxy · Control plane proxies provider and integration management.
- web-api → document-api: proxy · Control plane coordinates knowledge/document ingestion workflows.
- assistant-api → Pipeline: orchestrates · Assistant API routes live sessions into the realtime pipeline.
- assistant-api → Telephony: bridges · Assistant API connects phone systems into the assistant pipeline.
- assistant-api → WebRTC: streams · Assistant API serves browser-based realtime sessions.
- Pipeline → Audio core: processes · Pipeline consumes audio runtime components for low-latency speech handling.
- Pipeline → Transformers: uses · Pipeline delegates STT/TTS vendor work to transformers.
- Pipeline → LLM runtime: executes · Pipeline invokes model/agent execution and tool dispatch.
- LLM runtime → integration-api: calls · Assistant runtime uses normalized integration APIs for provider access.
- integration-api → Callers: implements · Provider gateway is backed by vendor-specific caller implementations.
- document-api → Doc pipeline: runs · Ingestion service executes the document processing pipeline.
- document-api → Connectors: ingests · Ingestion service pulls from source systems and storage backends.
- document-api → integration-api: calls · Document service uses shared integration/provider capabilities where needed.
- web-api → pkg: shares · Web API reuses shared auth, clients, and storage abstractions.
- assistant-api → pkg: shares · Assistant API consumes shared libraries for auth, clients, telemetry, and utilities.
- endpoint-api → pkg: shares · Endpoint API reuses common platform libraries.
- integration-api → pkg: shares · Integration API reuses common platform libraries.
- document-api → pkg: shares · Document service depends on shared clients and utilities.
- Cmd entrypoints → web-api: starts · Service binary launches the web API process.
- Cmd entrypoints → assistant-api: starts · Service binary launches the assistant API process.
- Cmd entrypoints → endpoint-api: starts · Service binary launches the endpoint API process.
- Cmd entrypoints → integration-api: starts · Service binary launches the integration API process.
- document-api → Infra: depends on · Ingestion relies on relational, cache, search, and object-storage infrastructure.
- assistant-api → Infra: depends on · Realtime assistant runtime depends on persistence, cache, search, and external providers.
- web-api → Infra: depends on · Control plane persists relational state and integrates with external services.
- endpoint-api → Infra: depends on · Endpoint backend stores definitions, versions, and logs in core infrastructure.
- integration-api → Infra: depends on · Provider gateway uses shared persistence and external vendor APIs.