rapidaai/voice-ai

1. This repo is a polyglot monorepo for a voice-AI platform, but the architectural center is the Go backend suite under api/, plus a React UI under ui/, and a separate Python document-ingestion service under api/document-api.

2. The main runtime boundaries are four Go services plus one Python service:

- web-api for product-facing CRUD and account/workspace management

- assistant-api for real-time agent orchestration and voice sessions

- endpoint-api for endpoint definition/invocation workflows

- integration-api for provider normalization and external AI integrations

- document-api for knowledge/document ingestion and indexing

These are deployed independently, with their own configs, migrations, and routers.

3. web-api is the control-plane backend for the UI. It manages organizations, projects, users, notifications, vault/connect resources, and proxy endpoints that fan out to the other APIs. It also owns its own relational schema via migrations and uses shared auth/service clients from pkg/.

4. assistant-api is the core data-plane service. It handles live voice sessions, assistant configuration, deployments, conversations, messages, telemetry, knowledge attachment, tools, and webhooks. Its architecture is layered: HTTP/gRPC-facing api/ handlers on top, service/use-case code in internal/services, and domain entities in internal/entity.

5. Inside assistant-api, the most important subsystem is the channel/pipeline layer: internal/channel/*, internal/adapters/*, and internal/channel/telephony/* / webrtc/*. This is where audio and session events are turned into a real-time pipeline that can accept input, dispatch it through STT/LLM/tooling, and emit output back to the channel.

6. The assistant voice pipeline depends on several low-level audio components:

- VAD engines in internal/vad

- noise suppression in internal/denoiser

- end-of-speech detectors in internal/end_of_speech

- audio resampling/normalization in internal/audio and internal/normalizer

These are not UI concerns; they are latency-critical runtime components that sit directly on the audio stream.

7. Telephony is a major boundary in assistant-api. The repo includes adapters for Twilio, Vonage, Telnyx, Exotel, SIP/Asterisk, and WebRTC, with media/session/registration state machines and outbound/inbound call handling. That means the platform is designed to bridge external phone systems into the same assistant pipeline used for web or debugger sessions.

8. Model/provider integration is abstracted heavily. api/assistant-api/internal/transformer/* contains provider-specific STT/TTS adapters for vendors like OpenAI, Deepgram, Google, Azure, AssemblyAI, ElevenLabs, Cartesia, NVIDIA, and others; api/integration-api/internal/caller/* contains the normalized chat/embedding/reranking/provider layer for LLM and related APIs. Together, these separate “provider specifics” from the higher-level assistant orchestration.

9. integration-api acts as a provider gateway. It exposes chat, embedding, reranking, audit logging, and unified provider endpoints, and internally it contains vendor-specific caller implementations and credential verification paths. This service is the main integration boundary between the platform and external AI vendors.

10. endpoint-api is the endpoint management/service-execution backend. It stores endpoint definitions, retry/cache configs, versions, provider models, and logs, then exposes invocation and trace retrieval. In the architecture diagram, this is the service for API-style endpoint execution rather than live voice sessions.

11. document-api is a separate Python pipeline for knowledge ingestion and indexing. It includes connectors to storage and source systems, extractors/chunkers/cleaners, vector-store and docstore components, Celery workers, and bridges to other services via generated protobuf clients. It depends on infrastructure like PostgreSQL, Redis, and OpenSearch, and is the service to draw for RAG/document processing rather than online voice execution.

12. The React ui/ package is the management console. It talks to web-api and proxy-backed APIs to manage assistants, endpoints, knowledge bases, integrations, users, and observability views. The UI is organized by domain pages and shared components, with many provider/config forms that mirror the backend’s provider abstractions.

13. Shared infrastructure and libraries live in pkg/ and are used across services. This includes:

- auth and scope authenticators

- generated/protobuf and REST clients

- configuration and connector factories

- telemetry/logging helpers

- storage abstractions

- plugin/provider registries

- common domain types and utilities

This layer is the main cross-service reuse boundary.

14. Persistence and external systems are central to the architecture: PostgreSQL for core relational state, Redis for caching/worker coordination, OpenSearch for document/vector search, and object storage backends like S3/GCS/Azure Blob for files and media assets. The services also integrate with many external SaaS/AI providers for speech, LLM, telephony, email, and telemetry.

15. Operationally, the repo is designed for containerized deployment. Each service has its own Dockerfile and env/config file, the root includes Docker Compose and nginx/API-gateway setup, and CI/CD workflows build images and deploy individual services. This supports both fully managed and self-hosted topologies.

16. For an architecture diagram, the clearest top-level picture is: UI and external clients hit the web/API gateway layer; that layer coordinates with assistant, endpoint, integration, and document services; assistant handles realtime audio/session execution via telephony/WebRTC and model providers; document-api handles ingestion/indexing; shared libs and infrastructure sit underneath all services.