# pixiv/chatvrm architecture

> Architecture diagram of the GitHub repository pixiv/chatvrm, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/pixiv/chatvrm
- Repository: https://github.com/pixiv/chatvrm

## Overview

##### Main Components
- Frontend/UI:
- This is built with Next.js (notice the “src/pages”, “src/components”, and design-related files like tailwind.config.js).
- Key subcomponents include:
∘ Chat Log (chatLog.tsx)
∘ Message Input (messageInput.tsx, messageInputContainer.tsx)
∘ 3D VRM Viewer (vrmViewer.tsx)
∘ Various UI pieces (menu, settings, iconButton, githubLink, etc.)
- API Endpoints:
- Located under “src/pages/api” with endpoints such as “chat.ts” (for ChatGPT API interactions) and “tts.ts” (for triggering text-to-speech via Koeiromap).
- Core Feature Modules:
- Chat functionality via “src/features/chat/openAiChat.ts”.
- Voice synthesis and message processing in “src/features/messages/” including voice synthesis and character speech.
- 3D character control modules such as the VRM viewer features (“src/features/vrmViewer”) and the various animation and control libraries under “src/lib” and “src/features/emoteController”.
- External Services:
- ChatGPT API for generating chat responses.
- Koeiromap API for voice synthesis.
- Web Speech API for speech recognition.
- @pixiv/three-vrm for rendering the 3D VRM model.

##### Relationships and Interactions
- The User Interface sends user input (chat messages) to the backend API endpoints.
- The backend “/api/chat” forwards the request to the ChatGPT API and returns generated responses.
- The “/api/tts” endpoint handles converting text to speech using the Koeiromap API, which is then relayed back to the UI.
- The 3D VRM viewer component interacts with VRM animation modules and controllers from “src/lib/VRMAnimation” and “src/features/emoteController” to render and animate the character based on the chat and voice input.
- Utility modules and constants (in “src/features/constants”, “src/utils”) support standardized data flow and transformation between components.

##### Architectural Patterns & Design Principles
- Next.js-based Structure: Clearly separates the client-side pages/components from the API endpoints.
- Modular Feature Separation: Organized “features”, “components”, “lib”, and “utils” indicate a component-based architecture with clear boundaries between UI, business logic, and external integrations.
- Integration with External APIs is abstracted into specific modules (i.e., chat, TTS, VRM animation), which makes the system extensible and maintainable.

##### Relevant Technologies
- Next.js, React, and TypeScript for the frontend.
- Three.js and @pixiv/three-vrm for 3D rendering.
- ChatGPT API and Koeiromap API for response generation and voice synthesis.
- Tailwind CSS (configured via tailwind.config.js) for styling.
- ESLint, postCSS, and other configuration tools that ensure code quality and build processes.

##### Elements
- Clear Labels: Each component should be labeled (e.g., “Chat Input Component”, “ChatGPT API Integration”, “3D VRM Viewer”, “Speech Recognition”).

##### Final Detailed Steps
- Under “Frontend,” include blocks for the 3D rendering (VRM Viewer) and interactive UI components.
- Under “Backend/API,” illustrate the API endpoints, then branch out to “External Services” blocks representing ChatGPT API and Koeiromap API.
- Integration of specialized libraries from the “features” and “lib” folders that drive animations (emote controller, lip sync, etc.).

## Diagram (Mermaid)

```mermaid
flowchart TD
    U["User"]:::user

    subgraph "Frontend UI"
        A["Main Application Pages"]:::frontend
        B["Chat Log Component"]:::frontend
        C["Message Input Components"]:::frontend
        D["3D VRM Viewer Component"]:::frontend
        subgraph "UI Auxiliary Components"
            E1["Menu"]:::frontend
            E2["Settings"]:::frontend
            E3["Icon Button"]:::frontend
            E4["GitHub Link"]:::frontend
        end
    end

    subgraph "Backend/API"
        F["ChatGPT API Endpoint"]:::backend
        G["TTS Endpoint"]:::backend
        H["Chat Functionality Module"]:::backend
        I["Voice Synthesis Module"]:::backend
        J["Character Speech Module"]:::backend
        K["Message Processing Module"]:::backend
        L["VRM Viewer Features"]:::backend
        subgraph "Animation & Emote Control"
            M1["Emote Controller"]:::backend
            M2["Lip Sync Module"]:::backend
        end
        subgraph "3D VRM Animation Libraries"
            N1["VRMAnimation"]:::backend
            N2["VRMLookAtSmoother"]:::backend
        end
        O["External Integration: Koeiromap API"]:::backend
        P["Utility & Constants Support"]:::backend
    end

    subgraph "External Services"
        Q["External ChatGPT API"]:::external
        R["External Koeiromap API"]:::external
        S["Web Speech API"]:::external
        T["3D Rendering: @pixiv/three-vrm"]:::external
    end

    %% Connections from User to Frontend UI
    U -->|"starts"| A
    U -->|"speaksVia"| S
    S -->|"transcribesTo"| C

    %% Frontend UI interactions
    A -->|"loads"| C
    C -->|"sendsChat"| F
    C -->|"sendsText"| G

    %% Backend Chat Flow
    F -->|"callsChatGPT"| Q
    Q -->|"returnsReply"| F
    F -->|"updatesChatLog"| B

    %% TTS Flow
    G -->|"triggersTTS"| R
    R -->|"returnsAudio"| G
    G -->|"audioFeedback"| B

    %% Chat Functionality Processing
    F -->|"processesMsg"| H
    H -->|"synthesizesVoice"| I
    H -->|"characterSpeech"| J
    H -->|"processesData"| K
    I -->|"usesUtilities"| P
    J -->|"usesUtilities"| P
    K -->|"usesUtilities"| P

    %% 3D Viewer and Animation Flow
    A -->|"renders3D"| D
    D -->|"usesViewerFeatures"| L
    L -->|"controlsAnimation"| M1
    L -->|"integratesLib"| N1
    L -->|"rendersWith"| T
    M1 -->|"syncsWith"| M2

    %% External Integration for Koeiromap API
    G -->|"utilizes"| O

    %% Click Events
    click A "https://github.com/pixiv/chatvrm/blob/main/src/pages/index.tsx"
    click B "https://github.com/pixiv/chatvrm/blob/main/src/components/chatLog.tsx"
    click C "https://github.com/pixiv/chatvrm/blob/main/src/components/messageInput.tsx"
    click D "https://github.com/pixiv/chatvrm/blob/main/src/components/vrmViewer.tsx"
    click E1 "https://github.com/pixiv/chatvrm/blob/main/src/components/menu.tsx"
    click E2 "https://github.com/pixiv/chatvrm/blob/main/src/components/settings.tsx"
    click E3 "https://github.com/pixiv/chatvrm/blob/main/src/components/iconButton.tsx"
    click E4 "https://github.com/pixiv/chatvrm/blob/main/src/components/githubLink.tsx"
    click F "https://github.com/pixiv/chatvrm/blob/main/src/pages/api/chat.ts"
    click G "https://github.com/pixiv/chatvrm/blob/main/src/pages/api/tts.ts"
    click H "https://github.com/pixiv/chatvrm/blob/main/src/features/chat/openAiChat.ts"
    click I "https://github.com/pixiv/chatvrm/blob/main/src/features/messages/synthesizeVoice.ts"
    click J "https://github.com/pixiv/chatvrm/blob/main/src/features/messages/speakCharacter.ts"
    click K "https://github.com/pixiv/chatvrm/blob/main/src/features/messages/messages.ts"
    click L "https://github.com/pixiv/chatvrm/blob/main/src/features/vrmViewer/viewer.ts"
    click M1 "https://github.com/pixiv/chatvrm/blob/main/src/features/emoteController/emoteController.ts"
    click M2 "https://github.com/pixiv/chatvrm/blob/main/src/features/lipSync/lipSync.ts"
    click N1 "https://github.com/pixiv/chatvrm/tree/main/src/lib/VRMAnimation"
    click N2 "https://github.com/pixiv/chatvrm/tree/main/src/lib/VRMLookAtSmootherLoaderPlugin"
    click O "https://github.com/pixiv/chatvrm/blob/main/src/features/koeiromap/koeiromap.ts"
    click P "https://github.com/pixiv/chatvrm/tree/main/src/utils"

    %% Styles
    classDef frontend fill:#a3d5ff,stroke:#1e90ff,stroke-width:2px;
    classDef backend fill:#ffdab9,stroke:#ff8c00,stroke-width:2px;
    classDef external fill:#d1e7dd,stroke:#198754,stroke-width:2px;
    classDef user fill:#f8d7da,stroke:#dc3545,stroke-width:2px;
```
