How socaity/py_audio2face works
Project type and purpose
• py_audio2face is an open-source Python client library/tool that drives NVIDIA’s Audio2Face headless server to generate facial animations from audio. It supports both file-based workflows (via a REST API) and real-time streaming (via gRPC).
File structure
• Top-level:
• py_audio2face/: the main package
• tests/: unit tests for core functionality
• pyproject.toml, MANIFEST.in: packaging/config
• Inside py_audio2face/:
• audio2face.py: entry point, Audio2Face class with high-level methods
• settings.py, utils.py: global settings and helpers
• modules/: encapsulates functional sub-systems
– _general.py: generic server control (start, shutdown)
– _audio2emotion.py: emotion detection logic
– _export.py: post-processing and USD export
– _player.py: playback controls (REST)
– _streaming.py: streaming via gRPC
– clients/: HTTP & gRPC stubs (requests-based HTTP client + autogenerated protobuf stubs)
• assets/: bundled USD scene templates (e.g., mark_arkit_solved_default.usd)
README for additional insights
• Describes both “single‐file” and “folder” batch processing via a2f.audio2face_single/folder
• Describes emotion configuration and streaming usage via media-toolkit and gRPC
• Highlights use cases: game-character, virtual avatars, USD export, real-time animation
a. Main components
• Client Application (your Python script importing py_audio2face)
• py_audio2face Library
– High-level API (Audio2Face class)
– Core modules (_general, _audio2emotion, _export, _player, _streaming)
– HTTP Client module
– gRPC Client stubs
– Local assets (USD scenes)
• Audio2Face Headless Server
– REST API endpoint
– gRPC endpoint for streaming
• File System
– Input: audio files or audio data stream
– Output: USD animation files
• Optional: Media-Toolkit (audio generator) or other audio sources
• Downstream consumers (Maya/Unreal Engine)
b. Relationships and interactions
1. User script → calls py_audio2face.Audio2Face methods
2. Audio2Face.init() → loads a USD scene (assets) on the remote server
3. For file mode: HTTP POST audio file path → server processes → REST GET to fetch animation → _export writes USD
4. For streaming mode: establish gRPC channel → stream audio frames via gRPC client stub → server streams blendshape frames back → _export writes USD or pushes frames onward
5. Emotion sub-module may analyze audio via local logic or remote service call before or during streaming
c. Architectural patterns & design principles
• Modular layering: separation of concerns between control (general), emotion, export, playback, streaming
• Adapter pattern: HTTP client and gRPC stub modules abstract transport details
• Template reuse: USD scene templates in assets folder
• Configurable settings: centralized in settings.py
d. Key technologies/frameworks
• Python 3.x
• requests (HTTP client)
• grpcio, protobuf (gRPC streaming)
• USD (Universal Scene Description) for animation export
• NVIDIA Audio2Face REST & gRPC APIs
Putting it all together
1. Top: “User Application” calls into
2. Middle: “py_audio2face Library” subdivided into five modules + clients + assets
3. Right: “Audio2Face Headless Server” with two APIs (REST, gRPC)
4. Bottom: “File System” (audio in, USD out)
• Key flows—single file mode vs streaming mode—and note where emotion detection, export, and playback modules intervene.