How pixiv/chatvrm works
Main Components
• Frontend/UI:
• This is built with Next.js (notice the “src/pages”, “src/components”, and design-related files like tailwind.config.js).
• Key subcomponents include:
∘ Chat Log (chatLog.tsx)
∘ Message Input (messageInput.tsx, messageInputContainer.tsx)
∘ 3D VRM Viewer (vrmViewer.tsx)
∘ Various UI pieces (menu, settings, iconButton, githubLink, etc.)
• API Endpoints:
• Located under “src/pages/api” with endpoints such as “chat.ts” (for ChatGPT API interactions) and “tts.ts” (for triggering text-to-speech via Koeiromap).
• Core Feature Modules:
• Chat functionality via “src/features/chat/openAiChat.ts”.
• Voice synthesis and message processing in “src/features/messages/” including voice synthesis and character speech.
• 3D character control modules such as the VRM viewer features (“src/features/vrmViewer”) and the various animation and control libraries under “src/lib” and “src/features/emoteController”.
• External Services:
• ChatGPT API for generating chat responses.
• Koeiromap API for voice synthesis.
• Web Speech API for speech recognition.
• @pixiv/three-vrm for rendering the 3D VRM model.
Relationships and Interactions
• The User Interface sends user input (chat messages) to the backend API endpoints.
• The backend “/api/chat” forwards the request to the ChatGPT API and returns generated responses.
• The “/api/tts” endpoint handles converting text to speech using the Koeiromap API, which is then relayed back to the UI.
• The 3D VRM viewer component interacts with VRM animation modules and controllers from “src/lib/VRMAnimation” and “src/features/emoteController” to render and animate the character based on the chat and voice input.
• Utility modules and constants (in “src/features/constants”, “src/utils”) support standardized data flow and transformation between components.
Architectural Patterns & Design Principles
• Next.js-based Structure: Clearly separates the client-side pages/components from the API endpoints.
• Modular Feature Separation: Organized “features”, “components”, “lib”, and “utils” indicate a component-based architecture with clear boundaries between UI, business logic, and external integrations.
• Integration with External APIs is abstracted into specific modules (i.e., chat, TTS, VRM animation), which makes the system extensible and maintainable.
Relevant Technologies
• Next.js, React, and TypeScript for the frontend.
• Three.js and @pixiv/three-vrm for 3D rendering.
• ChatGPT API and Koeiromap API for response generation and voice synthesis.
• Tailwind CSS (configured via tailwind.config.js) for styling.
• ESLint, postCSS, and other configuration tools that ensure code quality and build processes.
Elements
• Clear Labels: Each component should be labeled (e.g., “Chat Input Component”, “ChatGPT API Integration”, “3D VRM Viewer”, “Speech Recognition”).
Final Detailed Steps
• Under “Frontend,” include blocks for the 3D rendering (VRM Viewer) and interactive UI components.
• Under “Backend/API,” illustrate the API endpoints, then branch out to “External Services” blocks representing ChatGPT API and Koeiromap API.
• Integration of specialized libraries from the “features” and “lib” folders that drive animations (emote controller, lip sync, etc.).