How nvidia-ai-blueprints/digital-human works
Main Components
• Digital Avatar Pipeline – This part handles 3D animation, speech synthesis, rendering, and real‐time interactions. It uses NVIDIA Tokkio technologies (e.g., Audio2Face for avatar animation, Omniverse RTX for rendering, and integrations with Riva/ElevenLabs for speech).
• RAG Pipeline – This component is responsible for processing natural language queries using LLMs and generative models. It takes care of question-answer interactions and generates the conversation context necessary for the digital human.
• API / Integration Layer – The two pipelines (digital avatar and RAG) communicate via REST APIs. Look at the “api” directory (containing the openapi.json) to define the endpoints, request/response structures, and integration points.
• Deployment & Configuration – The “deploy” directory contains various Docker Compose files, environment scripts (print_env.sh, data_loader.sh), and configuration YAML files used to automate deployment in different environments and variations of the workflow.
• Customization and Evaluation – The “customize” and “evaluate” directories indicate areas where developers can adjust the pipeline (e.g., avatar customization and RAG fine-tuning) and evaluate the system using Jupyter notebooks and additional scripts.
Architectural Patterns and Technologies
• Separation of concerns: the digital avatar component is distinct from the RAG pipeline, and they communicate using REST – this reflects a modular microservices style.
• External services like the NVIDIA NIM inference microservices and third-party APIs (e.g., ElevenLabs, Riva) that power parts of the digital human application.
• That deployment is containerized; include Docker Compose (multiple variations for different microservices configurations) and clearly mention that the project supports both digital avatar and customized RAG deployments.
• From there, “drill down” further by adding sub-modules where necessary – note customization options and evaluation components.