# brightprogrammer7/ai-avatar-chatbot architecture

> Architecture diagram of the GitHub repository brightprogrammer7/ai-avatar-chatbot, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/brightprogrammer7/ai-avatar-chatbot
- Repository: https://github.com/brightprogrammer7/ai-avatar-chatbot

## Overview

##### Main Components
- Frontend: This is a web-based client built with modern JavaScript (React/JSX), using Vite for bundling and Tailwind CSS for styling. It hosts the user interface components (Avatar, ChatInterface, Scenario) and manages user interactions.
- Backend: A Node.js server that orchestrates the conversation. It includes:
- API modules for generating responses (using OpenAI’s GPT model),
- Audio generation using Eleven Labs,
- Lip-syncing via Rhubarb Lip Sync,
- Speech-to-Text processing leveraging OpenAI’s Whisper.
- External Services: OpenAI (for GPT and Whisper), Eleven Labs (for TTS), and the Rhubarb Lip Sync tool (installed locally and integrated via backend modules).
- Data & Assets: Audio files (wav/json) and 3D animation/model files stored in the apps/** directories and the resources folder for static assets.

##### Relationships and Interactions
- User Inputs:
- Text Input: The user sends a text message from the frontend, which is forwarded to the backend.
- Audio Input: The user submits audio, which is processed by Whisper for transcription.
- Backend Workflow:
- For text-based flows:
a. The backend receives text input.
b. It then uses the defined ChatPromptTemplate to create a conversation (with JSON output specifying text, facial expression, and animation).
c. The processed response is forwarded to Eleven Labs to generate TTS audio.
d. The audio is then passed to Rhubarb Lip Sync for viseme (lip movement) metadata generation.
- For audio-based flows:
a. The backend uses Whisper to convert the audio into text.
b. The converted text goes through the same steps as the text workflow.
- Frontend Interaction:
- The Frontend consumes the processed JSON (which includes text, facial expression, animation, audio, and lip-sync metadata).
- The user interface synchronizes the avatar’s animations and facial expressions with the generated audio and viseme data, providing a realistic digital human interaction.

##### Architectural Patterns and Technologies
- Modular Design: The backend is divided into modules (defaultMessages, elevenLabs, lip-sync, openAI, rhubarbLipSync, whisper) that clearly separate concerns.
- API Chaining: Notice how responses flow sequentially — prompt creation → model processing → structured parsing — using a chain of operations (as evident in the code snippet).
- Monorepo Structure: With separate directories for apps/backend and apps/frontend, it is easy to distinguish between client-side and server-side concerns.
- Use of Environment Variables: (with dotenv) to manage API keys and configuration for external services.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% User
    U("User"):::user

    %% Frontend Subgraph
    subgraph "Frontend"
        direction TB
        FE1("Avatar"):::frontend
        FE2("Chat Interface"):::frontend
        FE3("Scenario"):::frontend
    end

    %% Backend Subgraph
    subgraph "Backend"
        direction TB
        BE0("Backend Server"):::backend
        BE1("Speech-to-Text (Whisper)"):::backend
        BE2("Chat & Prompt Processing"):::backend
        BE3("Text-to-Speech (ElevenLabs)"):::backend
        BE4("Lip-sync"):::backend
        BE5("Rhubarb Lip Sync"):::backend
        BE6("Utility Functions"):::backend
    end

    %% External Services Subgraph
    subgraph "External Services"
        direction TB
        EX1("OpenAI API"):::external
        EX2("Eleven Labs API"):::external
        EX3("Rhubarb Lip Sync (Local)"):::external
    end

    %% Data & Assets Subgraph
    subgraph "Data & Assets"
        direction TB
        DA1("Animations"):::data
        DA2("Models"):::data
        DA3("Audio Files"):::data
    end

    %% Arrows - User to Frontend
    U -->|"interacts"| FE2

    %% Frontend to Data & Assets (Frontend loads visual resources)
    FE1 -->|"loads"| DA1
    FE1 -->|"loads"| DA2

    %% Frontend to Backend: Input submission from Chat Interface
    FE2 -->|"TextInput"| BE2
    FE2 -->|"AudioInput"| BE1

    %% Audio processing branch
    BE1 -->|"transcribes"| BE2

    %% Backend processing chain
    BE2 -->|"callsGPT"| EX1
    BE2 -->|"sendsPrompt"| BE3
    BE3 -->|"requestsTTS"| EX2
    BE3 -->|"producesAudio"| BE4
    BE4 -->|"preparesForLipsync"| BE5
    BE5 -->|"generatesViseme"| EX3

    %% Utility Functions for file and audio management used by Backend Server
    BE6 -->|"managesFiles"| DA3
    BE6 --> BE0

    %% Final output: Backend Server sends JSON to Frontend
    BE0 -->|"returnsJSON"| FE2

    %% Link Backend Server with processing modules
    BE0 --> BE1
    BE0 --> BE2
    BE0 --> BE3
    BE0 --> BE6

    %% Classes for styling
    classDef user fill:#ffe0b2,stroke:#bf360c,stroke-width:2px
    classDef frontend fill:#bbdefb,stroke:#0d47a1,stroke-width:2px
    classDef backend fill:#c8e6c9,stroke:#1b5e20,stroke-width:2px
    classDef external fill:#ffcc80,stroke:#ef6c00,stroke-width:2px
    classDef data fill:#e1bee7,stroke:#6a1b9a,stroke-width:2px

    %% Click Events for Frontend Components
    click FE1 "https://github.com/brightprogrammer7/ai-avatar-chatbot/blob/main/apps/frontend/src/components/Avatar.jsx"
    click FE2 "https://github.com/brightprogrammer7/ai-avatar-chatbot/blob/main/apps/frontend/src/components/ChatInterface.jsx"
    click FE3 "https://github.com/brightprogrammer7/ai-avatar-chatbot/blob/main/apps/frontend/src/components/Scenario.jsx"
    
    %% Click Events for Data & Assets - Frontend Assets & Audio Files
    click DA1 "https://github.com/brightprogrammer7/ai-avatar-chatbot/tree/main/apps/frontend/public/animations"
    click DA2 "https://github.com/brightprogrammer7/ai-avatar-chatbot/tree/main/apps/frontend/public/models"
    click DA3 "https://github.com/brightprogrammer7/ai-avatar-chatbot/tree/main/apps/backend/audios"

    %% Click Events for Backend Components
    click BE0 "https://github.com/brightprogrammer7/ai-avatar-chatbot/blob/main/apps/backend/server.js"
    click BE1 "https://github.com/brightprogrammer7/ai-avatar-chatbot/blob/main/apps/backend/modules/whisper.mjs"
    click BE2 "https://github.com/brightprogrammer7/ai-avatar-chatbot/blob/main/apps/backend/modules/openAI.mjs"
    click BE3 "https://github.com/brightprogrammer7/ai-avatar-chatbot/blob/main/apps/backend/modules/elevenLabs.mjs"
    click BE4 "https://github.com/brightprogrammer7/ai-avatar-chatbot/blob/main/apps/backend/modules/lip-sync.mjs"
    click BE5 "https://github.com/brightprogrammer7/ai-avatar-chatbot/blob/main/apps/backend/modules/rhubarbLipSync.mjs"
    click BE6 "https://github.com/brightprogrammer7/ai-avatar-chatbot/tree/main/apps/backend/utils"
```
