How uezo/chatdollkit works
Main Components
• 3D Model & Animation Subsystem – Represents the avatar and its controls. This includes components like the ModelController (which manages gestures, facial expressions, idle animations, and script-driven actions), the AIAvatar prefab, and associated animation registrations (as seen in the Demo and Scripts/Model directories).
• LLM (Large Language Model) Service Layer – Encapsulates the multiple AI dialog services (ChatGPT, Claude, Gemini, Dify, etc.) implemented in the Scripts/LLM folder. This layer handles the core conversation logic, system prompts, Chain of Thought processing, and function calling (tool integration).
• Speech Interface Components – This comprises two parts:
• Speech Listener (Speech-to-Text) – Controls input from the microphone using various implementations (e.g., Google, Azure, OpenAISpeechListener, etc. seen in Scripts/SpeechListener).
• Speech Synthesizer (Text-to-Speech) – Converts text responses to audio, handling multiple providers and modes (parallel, sequential) from Scripts/SpeechSynthesizer.
• External Communication & Control – This includes the Network, IO, and external messaging layers (socket and JavaScript message handlers under Scripts/Network and Plugins/), which allow external programs or web clients to control the avatar.
• UI and Input – Composed of UI components (prefabs for FPSManager, MicrophoneController, RequestInput, etc. in the Prefabs and Scripts/UI directories) that facilitate user interaction.
• Additional Tools/Extensions – Such as ChatMemory for long-term memory (under Extension/ChatMemory) and VRM-specific extensions present in Extension/VRM and Plugins.
Relationships and Data Flow
• At the top, the User interacts via UI and/or external clients. Their speech is recorded (via MicrophoneController and SpeechListener) and/or text entered.
• The Speech Listener passes recognized text to the Dialog Processor, which then passes it to the appropriate LLM Service.
• The LLM Service, after processing dialog context and optionally calling specific tools (via the AI Agent pattern), returns responses that may include tags for animations, expressions, and speech synthesizer actions.
• The Speech Synthesizer then converts the response text (with possible [pause:], [face:], [anim:] tags) into audio for playback.
• Concurrently, the ModelController interprets these tags and commands to trigger animations or facial expression changes on the 3D avatar.
• External control components (SocketServer and JavaScriptMessageHandler) allow this data flow to be influenced or initiated by third-party applications.
• Logging, debugging (e.g., FileLogger, FPSManager) and configuration (e.g., Editor scripts) support the system throughout.
Architectural Patterns
• Modularity: The code is split into clear “modules” (LLM, Speech, Model, UI, etc.) to support maintainability and customization.
• Event-Driven: Dialog and service components communicate via events (for example, handling user input or speech recognition events triggers dialog processing).
• Layered Architecture: User Interface/Pipelines → Processing (Speech Listener/Synthesizer, LLM/DLL communication) → 3D Model controls.
• Dependency and Plugin Injection: LLM services, speech providers, and external communication via plugins (WebGL JSLibs, etc.) indicate a dynamic plug-and-play design.