# punithvt/ai-avatar-system architecture

> Architecture diagram of the GitHub repository punithvt/ai-avatar-system, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-07-06). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/punithvt/ai-avatar-system
- Repository: https://github.com/punithvt/ai-avatar-system

## Overview

1. **High-level shape**
   - This is a three-tier system: a **Next.js frontend**, a **FastAPI backend**, and supporting **state/storage/worker infrastructure**.
   - The backend is the central orchestrator for both classic CRUD operations and the real-time avatar conversation pipeline.

2. **Frontend boundary**
   - The `frontend/` app is a Next.js 14 client with an App Router entrypoint and a componentized UI for **auth**, **avatar upload/listing**, **voice cloning**, **chat**, **history**, and **settings**.
   - Client state is managed with Zustand (`store/useStore.ts`) and server calls go through a shared API layer (`lib/api.ts`).
   - The frontend also consumes the backend WebSocket for live conversation updates and streamed avatar chunks.

3. **Backend API boundary**
   - `backend/app/api/v1/` exposes the main REST surface: `users`, `avatars`, `sessions`, `messages`, `voices`, and `conversations`.
   - These routes are the system’s control plane: registration/login, avatar CRUD, session lifecycle, message retrieval, and voice assignment/cloning.
   - The backend is a FastAPI app, so HTTP and WebSocket traffic share the same application runtime but different interaction patterns.

4. **Real-time conversation pipeline**
   - The most important runtime flow is the WebSocket session handler in `backend/app/websocket.py`.
   - A user sends text or audio; the backend transcribes audio with STT, feeds the text to an LLM, splits the response into chunks/sentences, synthesizes speech, then generates lip-synced video chunks for streaming back to the browser.
   - This is the core “turn-taking” loop and supports interruption/barge-in via a `stop` message.

5. **AI service layer**
   - `backend/app/services/` contains the pipeline’s main engines:
     - `stt.py` for speech-to-text,
     - `llm.py` for model/provider abstraction,
     - `tts.py` for speech synthesis and fallback logic,
     - `animator.py` for avatar/video generation,
     - `avatar_processor.py` for avatar asset preparation,
     - `storage.py` for file persistence,
     - `cache.py` for caching.
   - The architecture is intentionally modular here so the LLM, TTS, and avatar engine can be swapped or fall back independently.

6. **Persistence and domain model**
   - PostgreSQL is the primary durable store, with SQLAlchemy models in `backend/app/models.py` and API contracts in `backend/app/schemas.py`.
   - Alembic migrations live under `backend/alembic/`, showing that schema evolution is treated as a first-class part of the system.
   - The domain centers on users, avatars, voices, sessions, conversations, and messages, which align directly with the API routes and UI panels.

7. **Background processing**
   - Celery is wired in through `backend/app/celery_app.py`, with Redis acting as the broker/cache layer.
   - This separates latency-tolerant work from the request/response path, which is important for media processing, task offloading, and any asynchronous cleanup or generation steps.
   - The real-time WebSocket path is therefore distinct from slower worker-based jobs.

8. **Media and storage boundaries**
   - Avatar and voice assets move through a storage boundary handled by `storage.py`.
   - The project supports **local filesystem storage** for development and **S3/CloudFront-style remote storage** in production, which affects how uploads and generated media are served.
   - Nginx sits in front as the reverse proxy and is also the boundary for WebSocket forwarding in deployment.

9. **Security and request controls**
   - Authentication is JWT-based, with middleware support in `backend/app/middleware/security.py`.
   - Rate limiting is enforced separately in `backend/app/middleware/rate_limiter.py`.
   - This means the backend is designed for multi-user access, not a single-user demo, and the auth/session model is part of the core architecture.

10. **Observability and reliability**
   - Logging and telemetry are explicitly separated via `logging_config.py` and `telemetry.py`.
   - The repository includes tests for health, auth, users, avatars, sessions, conversations, WebSocket behavior, fallbacks, and end-to-end flows, which signals that the real-time pipeline is a major reliability concern.
   - CI/CD is defined in GitHub Actions, so build/test/release/deploy are part of the architecture rather than external scripts only.

11. **Deployment modes**
   - The repo supports local development with Docker Compose and a production path with GPU-aware containers, plus an AWS/Terraform path under `infrastructure/`.
   - `docker-compose.yml`, `docker-compose.prod.yml`, and `backend/Dockerfile` show that the backend can run CPU-only for development but is optimized for GPU-backed lip-sync generation in production.
   - `nginx/`, Terraform, and deploy scripts indicate multiple deployment topologies: local single-host, GPU VM, and managed cloud infrastructure.

12. **End-to-end data flow to diagram**
   - A good architecture diagram should show: **browser UI → REST/WebSocket → FastAPI orchestration → STT/LLM/TTS/animator services → Postgres/Redis/storage → streamed results back to the browser**.
   - The key distinction is that REST covers setup and persistence, while WebSocket carries the low-latency conversation stream and generated media chunks.

Main entry points: [`backend/app/websocket.py`](https://github.com/punithvt/ai-avatar-system/blob/main/backend/app/websocket.py) (WebSocket Gateway), [`backend/app/api/v1/__init__.py`](https://github.com/punithvt/ai-avatar-system/blob/main/backend/app/api/v1/__init__.py) (REST API), [`backend/app/services/storage.py`](https://github.com/punithvt/ai-avatar-system/blob/main/backend/app/services/storage.py) (Storage Service), [`nginx/nginx.conf`](https://github.com/punithvt/ai-avatar-system/blob/main/nginx/nginx.conf) (Nginx Proxy)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_frontend["Frontend"]
  node_browser(("Browser UI<br/>client<br/>[page.tsx]"))
  node_next_app["Next App<br/>nextjs app router<br/>[layout.tsx]"]
  node_ui_components["UI Panels<br/>component set<br/>[ChatInterface.tsx]"]
  node_api_client["API Client<br/>shared HTTP layer<br/>[api.ts]"]
  node_store["Client Store<br/>zustand state<br/>[useStore.ts]"]
end

subgraph group_backend["Backend"]
  node_rest_api["REST API<br/>fastapi routes<br/>[__init__.py]"]
  node_ws_gateway["WebSocket Gateway<br/>realtime session handler<br/>[websocket.py]"]
  node_auth_mw["Security Middleware<br/>jwt middleware<br/>[security.py]"]
  node_rate_mw["Rate Limiter<br/>middleware<br/>[rate_limiter.py]"]
  node_llm_service["LLM Engine<br/>model abstraction<br/>[llm.py]"]
  node_stt_service["Speech-to-Text<br/>audio transcription<br/>[stt.py]"]
  node_tts_service["Text-to-Speech<br/>speech synthesis<br/>[tts.py]"]
  node_animator_service["Avatar Animator<br/>lip-sync video<br/>[animator.py]"]
  node_storage_service["Storage Service<br/>file persistence<br/>[storage.py]"]
  node_avatar_processor["Avatar Prep<br/>media preprocessing"]
end

subgraph group_data["Data & Infra"]
  node_db[("Postgres<br/>primary datastore<br/>[database.py]")]
  node_redis[("Redis<br/>broker/cache<br/>[cache.py]")]
  node_celery["Celery Workers<br/>async jobs<br/>[celery_app.py]"]
  node_migrations["Alembic Migrations<br/>schema evolution"]
  node_nginx["Nginx Proxy<br/>reverse proxy<br/>[nginx.conf]"]
  node_infra["Terraform Infra<br/>cloud provisioning<br/>[main.tf]"]
end

node_browser -->|"renders"| node_next_app
node_next_app -->|"composes"| node_ui_components
node_ui_components -->|"reads state"| node_store
node_ui_components -->|"calls"| node_api_client
node_api_client -->|"REST"| node_rest_api
node_browser -->|"ws stream"| node_ws_gateway
node_ws_gateway -->|"audio→text"| node_stt_service
node_ws_gateway -->|"prompt"| node_llm_service
node_ws_gateway -->|"speech"| node_tts_service
node_ws_gateway -->|"video chunks"| node_animator_service
node_ws_gateway -->|"media I/O"| node_storage_service
node_ws_gateway -->|"session data"| node_db
node_rest_api -->|"protects"| node_auth_mw
node_rest_api -->|"limits"| node_rate_mw
node_rest_api -->|"crud"| node_db
node_rest_api -->|"uploads"| node_storage_service
node_avatar_processor -->|"prepares assets"| node_storage_service
node_storage_service -->|"cache hints"| node_redis
node_celery -->|"broker"| node_redis
node_celery -->|"background media"| node_storage_service
node_migrations -->|"evolves"| node_db
node_nginx -->|"serves"| node_next_app
node_nginx -->|"routes http"| node_rest_api
node_nginx -->|"routes ws"| node_ws_gateway
node_infra -->|"provisions"| node_nginx
node_infra -->|"provisions"| node_db

click node_browser "https://github.com/punithvt/ai-avatar-system/blob/main/frontend/app/page.tsx"
click node_next_app "https://github.com/punithvt/ai-avatar-system/blob/main/frontend/app/layout.tsx"
click node_ui_components "https://github.com/punithvt/ai-avatar-system/blob/main/frontend/components/ChatInterface.tsx"
click node_api_client "https://github.com/punithvt/ai-avatar-system/blob/main/frontend/lib/api.ts"
click node_store "https://github.com/punithvt/ai-avatar-system/blob/main/frontend/store/useStore.ts"
click node_rest_api "https://github.com/punithvt/ai-avatar-system/blob/main/backend/app/api/v1/__init__.py"
click node_ws_gateway "https://github.com/punithvt/ai-avatar-system/blob/main/backend/app/websocket.py"
click node_auth_mw "https://github.com/punithvt/ai-avatar-system/blob/main/backend/app/middleware/security.py"
click node_rate_mw "https://github.com/punithvt/ai-avatar-system/blob/main/backend/app/middleware/rate_limiter.py"
click node_llm_service "https://github.com/punithvt/ai-avatar-system/blob/main/backend/app/services/llm.py"
click node_stt_service "https://github.com/punithvt/ai-avatar-system/blob/main/backend/app/services/stt.py"
click node_tts_service "https://github.com/punithvt/ai-avatar-system/blob/main/backend/app/services/tts.py"
click node_animator_service "https://github.com/punithvt/ai-avatar-system/blob/main/backend/app/services/animator.py"
click node_storage_service "https://github.com/punithvt/ai-avatar-system/blob/main/backend/app/services/storage.py"
click node_avatar_processor "https://github.com/punithvt/ai-avatar-system/blob/main/backend/app/services/avatar_processor.py"
click node_db "https://github.com/punithvt/ai-avatar-system/blob/main/backend/app/database.py"
click node_redis "https://github.com/punithvt/ai-avatar-system/blob/main/backend/app/services/cache.py"
click node_celery "https://github.com/punithvt/ai-avatar-system/blob/main/backend/app/celery_app.py"
click node_migrations "https://github.com/punithvt/ai-avatar-system/blob/main/backend/alembic/versions/0000_initial_schema.py"
click node_nginx "https://github.com/punithvt/ai-avatar-system/blob/main/nginx/nginx.conf"
click node_infra "https://github.com/punithvt/ai-avatar-system/blob/main/infrastructure/main.tf"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_browser,node_next_app,node_ui_components,node_api_client,node_store toneBlue
class node_rest_api,node_ws_gateway,node_auth_mw,node_rate_mw,node_llm_service,node_stt_service,node_tts_service,node_animator_service,node_storage_service,node_avatar_processor toneAmber
class node_db,node_redis,node_celery,node_migrations,node_nginx,node_infra toneMint
```

## Components

### Frontend

Next.js client and UI state layer

- [Browser UI](https://github.com/punithvt/ai-avatar-system/blob/main/frontend/app/page.tsx): client · User-facing app for chat, avatars, and setup · `frontend/app/page.tsx`
- [Next App](https://github.com/punithvt/ai-avatar-system/blob/main/frontend/app/layout.tsx): nextjs app router · App Router shell and page composition · `frontend/app/layout.tsx`
- [UI Panels](https://github.com/punithvt/ai-avatar-system/blob/main/frontend/components/ChatInterface.tsx): component set · Auth, avatar, voice, chat, history, and settings screens · `frontend/components/ChatInterface.tsx`
- [API Client](https://github.com/punithvt/ai-avatar-system/blob/main/frontend/lib/api.ts): shared HTTP layer · Frontend wrapper for REST calls to the backend · `frontend/lib/api.ts`
- [Client Store](https://github.com/punithvt/ai-avatar-system/blob/main/frontend/store/useStore.ts): zustand state · Shared client state for conversation and UI status · `frontend/store/useStore.ts`

### Backend

FastAPI control plane and realtime runtime

- [REST API](https://github.com/punithvt/ai-avatar-system/blob/main/backend/app/api/v1/__init__.py): fastapi routes · Control plane for users, avatars, sessions, messages, voices, and conversations · `backend/app/api/v1/__init__.py`
- [WebSocket Gateway](https://github.com/punithvt/ai-avatar-system/blob/main/backend/app/websocket.py): realtime session handler · Low-latency conversation loop and interruption handling · `backend/app/websocket.py`
- [Security Middleware](https://github.com/punithvt/ai-avatar-system/blob/main/backend/app/middleware/security.py): jwt middleware · Authentication and request security checks · `backend/app/middleware/security.py`
- [Rate Limiter](https://github.com/punithvt/ai-avatar-system/blob/main/backend/app/middleware/rate_limiter.py): middleware · Per-user request control and abuse protection · `backend/app/middleware/rate_limiter.py`
- [LLM Engine](https://github.com/punithvt/ai-avatar-system/blob/main/backend/app/services/llm.py): model abstraction · Provider abstraction for generating assistant responses · `backend/app/services/llm.py`
- [Speech-to-Text](https://github.com/punithvt/ai-avatar-system/blob/main/backend/app/services/stt.py): audio transcription · Converts spoken input into text turns · `backend/app/services/stt.py`
- [Text-to-Speech](https://github.com/punithvt/ai-avatar-system/blob/main/backend/app/services/tts.py): speech synthesis · Generates voiced replies with fallback logic · `backend/app/services/tts.py`
- [Avatar Animator](https://github.com/punithvt/ai-avatar-system/blob/main/backend/app/services/animator.py): lip-sync video · Produces lip-synced avatar video chunks · `backend/app/services/animator.py`
- [Storage Service](https://github.com/punithvt/ai-avatar-system/blob/main/backend/app/services/storage.py): file persistence · Local or S3-style media storage boundary · `backend/app/services/storage.py`
- [Avatar Prep](https://github.com/punithvt/ai-avatar-system/blob/main/backend/app/services/avatar_processor.py): media preprocessing · Prepares uploaded avatar assets for generation · `backend/app/services/avatar_processor.py`

### Data & Infra

Persistence, cache, storage, and deployment boundary

- [Postgres](https://github.com/punithvt/ai-avatar-system/blob/main/backend/app/database.py): primary datastore · Durable storage for users, avatars, sessions, voices, conversations, and messages · `backend/app/database.py`
- [Redis](https://github.com/punithvt/ai-avatar-system/blob/main/backend/app/services/cache.py): broker/cache · Celery broker and caching layer · `backend/app/services/cache.py`
- [Celery Workers](https://github.com/punithvt/ai-avatar-system/blob/main/backend/app/celery_app.py): async jobs · Background execution for media and deferred tasks · `backend/app/celery_app.py`
- [Alembic Migrations](https://github.com/punithvt/ai-avatar-system/blob/main/backend/alembic/versions/0000_initial_schema.py): schema evolution · Database migration history and versioning · `backend/alembic/versions/0000_initial_schema.py`
- [Nginx Proxy](https://github.com/punithvt/ai-avatar-system/blob/main/nginx/nginx.conf): reverse proxy · Ingress boundary for HTTP and WebSocket traffic · `nginx/nginx.conf`
- [Terraform Infra](https://github.com/punithvt/ai-avatar-system/blob/main/infrastructure/main.tf): cloud provisioning · AWS-oriented infrastructure and deployment topology · `infrastructure/main.tf`

## Connections

- Browser UI → Next App: renders · Browser loads the Next.js app shell
- Next App → UI Panels: composes · App shell assembles feature panels
- UI Panels → Client Store: reads state · Panels consume shared client state
- UI Panels → API Client: calls · UI triggers REST operations through shared API layer
- API Client → REST API: REST · Frontend control-plane requests
- Browser UI → WebSocket Gateway: ws stream · Browser receives live conversation updates
- WebSocket Gateway → Speech-to-Text: audio→text · Transcribes spoken input
- WebSocket Gateway → LLM Engine: prompt · Generates assistant response text
- WebSocket Gateway → Text-to-Speech: speech · Synthesizes response audio
- WebSocket Gateway → Avatar Animator: video chunks · Creates lip-synced avatar output
- WebSocket Gateway → Storage Service: media I/O · Reads and writes avatar/media assets
- WebSocket Gateway → Postgres: session data · Persists conversation state and messages
- REST API → Security Middleware: protects · API routes are guarded by JWT security
- REST API → Rate Limiter: limits · Applies request throttling controls
- REST API → Postgres: crud · Reads and writes domain records
- REST API → Storage Service: uploads · Manages avatar and voice assets
- Avatar Prep → Storage Service: prepares assets · Transforms uploaded media before use
- Storage Service → Redis: cache hints · Caches or coordinates media lookup
- Celery Workers → Redis: broker · Asynchronous task dispatch and queueing
- Celery Workers → Storage Service: background media · Worker-side file and generation tasks
- Alembic Migrations → Postgres: evolves · Schema changes keep the database in sync
- Nginx Proxy → Next App: serves · Reverse proxy fronts the frontend app
- Nginx Proxy → REST API: routes http · Forwards API traffic to FastAPI
- Nginx Proxy → WebSocket Gateway: routes ws · Forwards WebSocket traffic to realtime backend
- Terraform Infra → Nginx Proxy: provisions · Infrastructure defines deployment boundary
- Terraform Infra → Postgres: provisions · Cloud resources include persistent datastore
