# uezo/chatdollkit architecture

> Architecture diagram of the GitHub repository uezo/chatdollkit, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-31). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/uezo/chatdollkit
- Repository: https://github.com/uezo/chatdollkit

## Overview

##### Main Components
- 3D Model & Animation Subsystem – Represents the avatar and its controls. This includes components like the ModelController (which manages gestures, facial expressions, idle animations, and script-driven actions), the AIAvatar prefab, and associated animation registrations (as seen in the Demo and Scripts/Model directories).
- LLM (Large Language Model) Service Layer – Encapsulates the multiple AI dialog services (ChatGPT, Claude, Gemini, Dify, etc.) implemented in the Scripts/LLM folder. This layer handles the core conversation logic, system prompts, Chain of Thought processing, and function calling (tool integration).
- Speech Interface Components – This comprises two parts:
- Speech Listener (Speech-to-Text) – Controls input from the microphone using various implementations (e.g., Google, Azure, OpenAISpeechListener, etc. seen in Scripts/SpeechListener).
- Speech Synthesizer (Text-to-Speech) – Converts text responses to audio, handling multiple providers and modes (parallel, sequential) from Scripts/SpeechSynthesizer.
- External Communication & Control – This includes the Network, IO, and external messaging layers (socket and JavaScript message handlers under Scripts/Network and Plugins/), which allow external programs or web clients to control the avatar.
- UI and Input – Composed of UI components (prefabs for FPSManager, MicrophoneController, RequestInput, etc. in the Prefabs and Scripts/UI directories) that facilitate user interaction.
- Additional Tools/Extensions – Such as ChatMemory for long-term memory (under Extension/ChatMemory) and VRM-specific extensions present in Extension/VRM and Plugins.

##### Relationships and Data Flow
- At the top, the User interacts via UI and/or external clients. Their speech is recorded (via MicrophoneController and SpeechListener) and/or text entered.
- The Speech Listener passes recognized text to the Dialog Processor, which then passes it to the appropriate LLM Service.
- The LLM Service, after processing dialog context and optionally calling specific tools (via the AI Agent pattern), returns responses that may include tags for animations, expressions, and speech synthesizer actions.
- The Speech Synthesizer then converts the response text (with possible [pause:], [face:], [anim:] tags) into audio for playback.
- Concurrently, the ModelController interprets these tags and commands to trigger animations or facial expression changes on the 3D avatar.
- External control components (SocketServer and JavaScriptMessageHandler) allow this data flow to be influenced or initiated by third-party applications.
- Logging, debugging (e.g., FileLogger, FPSManager) and configuration (e.g., Editor scripts) support the system throughout.

##### Architectural Patterns
- Modularity: The code is split into clear “modules” (LLM, Speech, Model, UI, etc.) to support maintainability and customization.
- Event-Driven: Dialog and service components communicate via events (for example, handling user input or speech recognition events triggers dialog processing).
- Layered Architecture: User Interface/Pipelines → Processing (Speech Listener/Synthesizer, LLM/DLL communication) → 3D Model controls.
- Dependency and Plugin Injection: LLM services, speech providers, and external communication via plugins (WebGL JSLibs, etc.) indicate a dynamic plug-and-play design.

## Diagram (Mermaid)

```mermaid
flowchart TD
    subgraph "User Interface Layer"
        UI["UI Components (FPSManager, MicrophoneController, RequestInput)"]:::ui
    end

    subgraph "Speech Interface Layer"
        SL["Speech Listener (Speech-to-Text)"]:::speech
        SS["Speech Synthesizer (Text-to-Speech)"]:::speech
    end

    subgraph "LLM Service Layer"
        LLM["LLM Services (ChatGPT,Claude,Gemini,Dify, etc.)"]:::llm
    end

    subgraph "3D Avatar & Model Control"
        MC["ModelController"]:::model
        AA["AI Avatar Prefab"]:::model
    end

    subgraph "External Communication & Control"
        EC["SocketServer/JavaScriptMessageHandler"]:::external
    end

    subgraph "Extensions"
        CM["ChatMemory"]:::extension
        VRM["VRM Extensions"]:::extension
    end

    %% Data Flow
    UI -->|"input"| SL
    SL -->|"recognizedText"| LLM
    LLM -->|"responseText"| SS
    LLM -->|"animationTags"| MC
    MC -->|"controlAnimation"| AA
    EC -->|"externalTrigger"| LLM
    EC -->|"externalUI"| UI

    %% Click Events
    click MC "https://github.com/uezo/chatdollkit/blob/master/Scripts/Model/ModelController.cs"
    click AA "https://github.com/uezo/chatdollkit/blob/master/Prefabs/AIAvatar.prefab"
    click LLM "https://github.com/uezo/chatdollkit/tree/master/Scripts/LLM"
    click SL "https://github.com/uezo/chatdollkit/tree/master/Scripts/SpeechListener"
    click SS "https://github.com/uezo/chatdollkit/tree/master/Scripts/SpeechSynthesizer"
    click EC "https://github.com/uezo/chatdollkit/tree/master/Scripts/Network"
    click UI "https://github.com/uezo/chatdollkit/tree/master/Scripts/UI"
    click CM "https://github.com/uezo/chatdollkit/tree/master/Extension/ChatMemory"
    click VRM "https://github.com/uezo/chatdollkit/tree/master/Extension/VRM"

    %% Styles
    classDef ui fill:#f9c74f,stroke:#333,stroke-width:2px;
    classDef speech fill:#90be6d,stroke:#333,stroke-width:2px;
    classDef llm fill:#f94144,stroke:#333,stroke-width:2px;
    classDef model fill:#577590,stroke:#333,stroke-width:2px;
    classDef external fill:#f8961e,stroke:#333,stroke-width:2px;
    classDef extension fill:#277da1,stroke:#333,stroke-width:2px;
```
