How uuembodiedsocialai/probtalk3d works
Main components of the system
• Data & Preprocessing
– Datasets: The “datasets” folder holds preprocessed 3DMEAD data (FLAME parameters, audio, various splits) and related scripts (in “pre_process” and “datasets/3DMEAD_preprocess”).
– Preprocessing Scripts: Scripts like pre_process/param_to_vert.py and dataset processing scripts in “datasets/3DMEAD_preprocess”.
• Configuration Management
– Hydra and YAML Files: The “configs” folder contains subdirectories for callback, data, model, optimizer, and machine configurations. These are used to control hyperparameters, training behavior, logging, etc.
• Core Framework
– Modular Code: The “framework” directory contains submodules for data loading, callbacks, launching tasks (with Blender integration), modeling (including feature extraction with HuBERT, motion encoder/decoder modules, losses, metrics, and more). This is the backbone that wires together data processing, training, and inference.
• Model Training Pipeline
– Stage 1 (Motion Prior): Training the VQ-VAE (or VAE variant) to learn a motion prior. Executable through scripts like train_vqvae_prior.sh and commands in train_all.py with experiment set to “vqvae_prior” or “vae_prior.”
– Stage 2 (Prediction): This stage uses the learned prior to condition a speech and emotion-based network, configured via train_all.py (experiment = vqvae_pred/vae_pred), with parameters pointing to the stage 1 checkpoint.
• Evaluation & Generation
– Evaluation: Includes scripts such as evaluation.py and evaluation_quality.py, with command-line overrides to resume or set GPU IDs. There’s also support for running via Slurm (evaluation.sh).
– Generation: The generation.py script takes test audio and produces predicted FLAME parameters (.npy files). Further, there is a rendering pipeline (render_param.py, render_vert.py) that converts the predictions into videos via Blender.
• Comparison Branch – Diffusion Model (FaceDiffuser)
– “diffusion” Folder: Contains an alternative model implementation (and related evaluation/generation scripts) for comparing with the VQ-VAE approach.
• External Dependencies / Reused Code
– “deps” Folder: Contains re-implemented and extended code for FLAME, HuBERT, and other borrowed modules. These represent the external libraries or adapted code contributing to the model’s features.
– “static” Folder: Likely for the project website or UI elements that could be referenced during demonstration (CSS, JS, images).
Relationships and Interactions
• Data flows from raw datasets (3DMEAD) through preprocessing scripts into the “framework” modules.
• The training pipeline has a clear sequence: data loading (using configuration settings) ⟶ feature extraction (using HuBERT and other modules) ⟶ model forward passes (using different submodules like encoders, decoders, and loss functions) ⟶ checkpoint saving. Note the bifurcation for Stage 1 (prior training) and Stage 2 (prediction training using the prior).
• Evaluation and generation modules use the trained checkpoints to generate outputs; these outputs are processed and rendered.
• The diffusion branch (in its own directory) interacts in a similar way but is designed for a slightly different technical process (it implements a diffusion-based generative model for 3D facial animation).
Architectural Patterns and Design Principles
• Modular Design: The project is split into self-contained modules (data, model, tools, evaluation) to facilitate experimentation.
• Hydra-based Configuration: The use of Hydra allows dynamic configuration overrides, making the system flexible for different training or inference scenarios.
• Two-Stage Training Pipeline: Clear separation between learning a motion prior and conditioning on speech/emotion for final predictions.
• Extensibility: With separate directories for external dependencies and alternative approaches (diffusion/facediffuser), the architecture is built to be extended or swapped out.