How punithvt/ai-avatar-system works
1. High-level shape
- This is a three-tier system: a Next.js frontend, a FastAPI backend, and supporting state/storage/worker infrastructure.
- The backend is the central orchestrator for both classic CRUD operations and the real-time avatar conversation pipeline.
2. Frontend boundary
- The frontend/ app is a Next.js 14 client with an App Router entrypoint and a componentized UI for auth, avatar upload/listing, voice cloning, chat, history, and settings.
- Client state is managed with Zustand (store/useStore.ts) and server calls go through a shared API layer (lib/api.ts).
- The frontend also consumes the backend WebSocket for live conversation updates and streamed avatar chunks.
3. Backend API boundary
- backend/app/api/v1/ exposes the main REST surface: users, avatars, sessions, messages, voices, and conversations.
- These routes are the system’s control plane: registration/login, avatar CRUD, session lifecycle, message retrieval, and voice assignment/cloning.
- The backend is a FastAPI app, so HTTP and WebSocket traffic share the same application runtime but different interaction patterns.
4. Real-time conversation pipeline
- The most important runtime flow is the WebSocket session handler in backend/app/websocket.py.
- A user sends text or audio; the backend transcribes audio with STT, feeds the text to an LLM, splits the response into chunks/sentences, synthesizes speech, then generates lip-synced video chunks for streaming back to the browser.
- This is the core “turn-taking” loop and supports interruption/barge-in via a stop message.
5. AI service layer
- backend/app/services/ contains the pipeline’s main engines:
- stt.py for speech-to-text,
- llm.py for model/provider abstraction,
- tts.py for speech synthesis and fallback logic,
- animator.py for avatar/video generation,
- avatar_processor.py for avatar asset preparation,
- storage.py for file persistence,
- cache.py for caching.
- The architecture is intentionally modular here so the LLM, TTS, and avatar engine can be swapped or fall back independently.
6. Persistence and domain model
- PostgreSQL is the primary durable store, with SQLAlchemy models in backend/app/models.py and API contracts in backend/app/schemas.py.
- Alembic migrations live under backend/alembic/, showing that schema evolution is treated as a first-class part of the system.
- The domain centers on users, avatars, voices, sessions, conversations, and messages, which align directly with the API routes and UI panels.
7. Background processing
- Celery is wired in through backend/app/celery_app.py, with Redis acting as the broker/cache layer.
- This separates latency-tolerant work from the request/response path, which is important for media processing, task offloading, and any asynchronous cleanup or generation steps.
- The real-time WebSocket path is therefore distinct from slower worker-based jobs.
8. Media and storage boundaries
- Avatar and voice assets move through a storage boundary handled by storage.py.
- The project supports local filesystem storage for development and S3/CloudFront-style remote storage in production, which affects how uploads and generated media are served.
- Nginx sits in front as the reverse proxy and is also the boundary for WebSocket forwarding in deployment.
9. Security and request controls
- Authentication is JWT-based, with middleware support in backend/app/middleware/security.py.
- Rate limiting is enforced separately in backend/app/middleware/rate_limiter.py.
- This means the backend is designed for multi-user access, not a single-user demo, and the auth/session model is part of the core architecture.
10. Observability and reliability
- Logging and telemetry are explicitly separated via logging_config.py and telemetry.py.
- The repository includes tests for health, auth, users, avatars, sessions, conversations, WebSocket behavior, fallbacks, and end-to-end flows, which signals that the real-time pipeline is a major reliability concern.
- CI/CD is defined in GitHub Actions, so build/test/release/deploy are part of the architecture rather than external scripts only.
11. Deployment modes
- The repo supports local development with Docker Compose and a production path with GPU-aware containers, plus an AWS/Terraform path under infrastructure/.
- docker-compose.yml, docker-compose.prod.yml, and backend/Dockerfile show that the backend can run CPU-only for development but is optimized for GPU-backed lip-sync generation in production.
- nginx/, Terraform, and deploy scripts indicate multiple deployment topologies: local single-host, GPU VM, and managed cloud infrastructure.
12. End-to-end data flow to diagram
- A good architecture diagram should show: browser UI → REST/WebSocket → FastAPI orchestration → STT/LLM/TTS/animator services → Postgres/Redis/storage → streamed results back to the browser.
- The key distinction is that REST covers setup and persistence, while WebSocket carries the low-latency conversation stream and generated media chunks.
Main entry points: websocket.py WebSocket Gateway · __init__.py REST API · storage.py Storage Service · nginx.conf Nginx Proxy
Components 21
Frontend
Next.js client and UI state layer
- Browser UI · User-facing app for chat, avatars, and setup
frontend/app/page.tsx - Next App · App Router shell and page composition
frontend/app/layout.tsx - UI Panels · Auth, avatar, voice, chat, history, and settings screens
frontend/components/ChatInterface.tsx - API Client · Frontend wrapper for REST calls to the backend
frontend/lib/api.ts - Client Store · Shared client state for conversation and UI status
frontend/store/useStore.ts
Backend
FastAPI control plane and realtime runtime
- REST API · Control plane for users, avatars, sessions, messages, voices, and conversations
backend/app/api/v1/__init__.py - WebSocket Gateway · Low-latency conversation loop and interruption handling
backend/app/websocket.py - Security Middleware · Authentication and request security checks
backend/app/middleware/security.py - Rate Limiter · Per-user request control and abuse protection
backend/app/middleware/rate_limiter.py - LLM Engine · Provider abstraction for generating assistant responses
backend/app/services/llm.py - Speech-to-Text · Converts spoken input into text turns
backend/app/services/stt.py - Text-to-Speech · Generates voiced replies with fallback logic
backend/app/services/tts.py - Avatar Animator · Produces lip-synced avatar video chunks
backend/app/services/animator.py - Storage Service · Local or S3-style media storage boundary
backend/app/services/storage.py - Avatar Prep · Prepares uploaded avatar assets for generation
backend/app/services/avatar_processor.py
Data & Infra
Persistence, cache, storage, and deployment boundary
- Postgres · Durable storage for users, avatars, sessions, voices, conversations, and messages
backend/app/database.py - Redis · Celery broker and caching layer
backend/app/services/cache.py - Celery Workers · Background execution for media and deferred tasks
backend/app/celery_app.py - Alembic Migrations · Database migration history and versioning
backend/alembic/versions/0000_initial_schema.py - Nginx Proxy · Ingress boundary for HTTP and WebSocket traffic
nginx/nginx.conf - Terraform Infra · AWS-oriented infrastructure and deployment topology
infrastructure/main.tf
Connections 26
- Browser UI → Next App: renders · Browser loads the Next.js app shell
- Next App → UI Panels: composes · App shell assembles feature panels
- UI Panels → Client Store: reads state · Panels consume shared client state
- UI Panels → API Client: calls · UI triggers REST operations through shared API layer
- API Client → REST API: REST · Frontend control-plane requests
- Browser UI → WebSocket Gateway: ws stream · Browser receives live conversation updates
- WebSocket Gateway → Speech-to-Text: audio→text · Transcribes spoken input
- WebSocket Gateway → LLM Engine: prompt · Generates assistant response text
- WebSocket Gateway → Text-to-Speech: speech · Synthesizes response audio
- WebSocket Gateway → Avatar Animator: video chunks · Creates lip-synced avatar output
- WebSocket Gateway → Storage Service: media I/O · Reads and writes avatar/media assets
- WebSocket Gateway → Postgres: session data · Persists conversation state and messages
- REST API → Security Middleware: protects · API routes are guarded by JWT security
- REST API → Rate Limiter: limits · Applies request throttling controls
- REST API → Postgres: crud · Reads and writes domain records
- REST API → Storage Service: uploads · Manages avatar and voice assets
- Avatar Prep → Storage Service: prepares assets · Transforms uploaded media before use
- Storage Service → Redis: cache hints · Caches or coordinates media lookup
- Celery Workers → Redis: broker · Asynchronous task dispatch and queueing
- Celery Workers → Storage Service: background media · Worker-side file and generation tasks
- Alembic Migrations → Postgres: evolves · Schema changes keep the database in sync
- Nginx Proxy → Next App: serves · Reverse proxy fronts the frontend app
- Nginx Proxy → REST API: routes http · Forwards API traffic to FastAPI
- Nginx Proxy → WebSocket Gateway: routes ws · Forwards WebSocket traffic to realtime backend
- Terraform Infra → Nginx Proxy: provisions · Infrastructure defines deployment boundary
- Terraform Infra → Postgres: provisions · Cloud resources include persistent datastore