How ubisoft/ubisoft-laforge-zeroeggs works
Project Type & Purpose
• This is a research / experimental full-stack project dealing with gesture generation from speech. In essence, it is an end‐to‐end system that processes raw audio and motion capture (BVH) data, trains a model to learn examples of speech-to-gesture mapping, and finally runs inference to generate gesture animations.
• The project also features a data pipeline, training/inference routines, and even a rendering toolchain (bvh2fbx) for conversion to FBX files intended for integration with motion editing/rendering software.
File Structure
• The major directories/components are:
• The “ZEGGS” package: This contains the main Python modules for the project. Within it:
• The “anim” subfolder deals with BVH file formats and transformations (files like bvh.py, xform.py, etc.).
• The “audio” subfolder handles audio processing including signal manipulation, logs, and spectrogram generation.
• Modules for training (train.py, main.py) and data preparation (data_pipeline.py) are provided as well as utilities and optimizers.
• “configs” holds configuration files (JSON) for different versions (v1, v2) and for the data pipeline.
• “data” includes the large raw dataset (with multipart zip files), processed data (processed_v1 and processed_v2), outputs from training (saved model files), and test/evaluation files.
• “bvh2fbx” contains scripts and assets for converting BVH animation files to FBX using external tools like MotionBuilder.
• “media” and “README.md” provide visual references and documentation.
• This structure reveals a modular architecture divided by functional responsibilities (data processing, training/inference, file conversion).
Core Components and Their Relationships
• Data Preparation Pipeline:
• Raw data (zipped dataset) → Extraction → data_pipeline.py reading configuration (via configs/*.json) → Processed data (stored in data/processed_v1 and processed_v2)
• Model Training:
• The training routines (handled primarily by main.py, train.py) pick up processed data and configuration settings; they train the gesture generation model and output saved models into data/outputs.
• Inference / Gesture Generation:
• generate.py, together with command-line interfaces (CLI), accepts input audio files and style files, uses pre-trained models, and produces gesture animation outputs in BVH format.
• Rendering:
• The bvh2fbx module takes the generated BVH files and translates them into FBX files (with provided character models); note that both a Python script and a batch script are provided to bridge with MotionBuilder.
• Support Modules:
• The “anim” and “audio” subpackages serve as libraries that define transformation routines, mathematical helpers, and utility functions used across training, inference, and evaluation.
• External Resources/Dependencies:
• Environment setup details (Python 3.8, CUDA, PyTorch) indicate deep learning components; integration with external tools for audio (such as sox) and rendering (MotionBuilder) also plays a role.
Architectural Patterns & Design Principles
• Modularity: Clear separation of concerns is visible via separate directories for animation, audio, and data processing. This modular design promotes maintainability and extensibility.
• Configurability: The use of external JSON configuration files for both training and data pipeline steps suggests a flexible, parameter-driven design.
• Pipeline-based Processing: Data flows sequentially from raw extraction → processing → training → inference, and finally to rendering conversion.
• Reusability: Core helper and utility functions (helpers.py, utils.py) support multiple stages of the workflow.
• CLI and Batch Scripting: The project supports both interactive (CLI-based) and batch-processing modes, which is common for research prototypes.