How brightprogrammer7/ai-avatar-chatbot works
Main Components
• Frontend: This is a web-based client built with modern JavaScript (React/JSX), using Vite for bundling and Tailwind CSS for styling. It hosts the user interface components (Avatar, ChatInterface, Scenario) and manages user interactions.
• Backend: A Node.js server that orchestrates the conversation. It includes:
• API modules for generating responses (using OpenAI’s GPT model),
• Audio generation using Eleven Labs,
• Lip-syncing via Rhubarb Lip Sync,
• Speech-to-Text processing leveraging OpenAI’s Whisper.
• External Services: OpenAI (for GPT and Whisper), Eleven Labs (for TTS), and the Rhubarb Lip Sync tool (installed locally and integrated via backend modules).
• Data & Assets: Audio files (wav/json) and 3D animation/model files stored in the apps/** directories and the resources folder for static assets.
Relationships and Interactions
• User Inputs:
• Text Input: The user sends a text message from the frontend, which is forwarded to the backend.
• Audio Input: The user submits audio, which is processed by Whisper for transcription.
• Backend Workflow:
• For text-based flows:
a. The backend receives text input.
b. It then uses the defined ChatPromptTemplate to create a conversation (with JSON output specifying text, facial expression, and animation).
c. The processed response is forwarded to Eleven Labs to generate TTS audio.
d. The audio is then passed to Rhubarb Lip Sync for viseme (lip movement) metadata generation.
• For audio-based flows:
a. The backend uses Whisper to convert the audio into text.
b. The converted text goes through the same steps as the text workflow.
• Frontend Interaction:
• The Frontend consumes the processed JSON (which includes text, facial expression, animation, audio, and lip-sync metadata).
• The user interface synchronizes the avatar’s animations and facial expressions with the generated audio and viseme data, providing a realistic digital human interaction.
Architectural Patterns and Technologies
• Modular Design: The backend is divided into modules (defaultMessages, elevenLabs, lip-sync, openAI, rhubarbLipSync, whisper) that clearly separate concerns.
• API Chaining: Notice how responses flow sequentially — prompt creation → model processing → structured parsing — using a chain of operations (as evident in the code snippet).
• Monorepo Structure: With separate directories for apps/backend and apps/frontend, it is easy to distinguish between client-side and server-side concerns.
• Use of Environment Variables: (with dotenv) to manage API keys and configuration for external services.