# uuembodiedsocialai/probtalk3d architecture

> Architecture diagram of the GitHub repository uuembodiedsocialai/probtalk3d, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-01). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/uuembodiedsocialai/probtalk3d
- Repository: https://github.com/uuembodiedsocialai/probtalk3d

## Overview

##### Main components of the system
- Data & Preprocessing
– Datasets: The “datasets” folder holds preprocessed 3DMEAD data (FLAME parameters, audio, various splits) and related scripts (in “pre_process” and “datasets/3DMEAD_preprocess”).
– Preprocessing Scripts: Scripts like pre_process/param_to_vert.py and dataset processing scripts in “datasets/3DMEAD_preprocess”.
- Configuration Management
– Hydra and YAML Files: The “configs” folder contains subdirectories for callback, data, model, optimizer, and machine configurations. These are used to control hyperparameters, training behavior, logging, etc.
- Core Framework
– Modular Code: The “framework” directory contains submodules for data loading, callbacks, launching tasks (with Blender integration), modeling (including feature extraction with HuBERT, motion encoder/decoder modules, losses, metrics, and more). This is the backbone that wires together data processing, training, and inference.
- Model Training Pipeline
– Stage 1 (Motion Prior): Training the VQ-VAE (or VAE variant) to learn a motion prior. Executable through scripts like train_vqvae_prior.sh and commands in train_all.py with experiment set to “vqvae_prior” or “vae_prior.”
– Stage 2 (Prediction): This stage uses the learned prior to condition a speech and emotion-based network, configured via train_all.py (experiment = vqvae_pred/vae_pred), with parameters pointing to the stage 1 checkpoint.
- Evaluation & Generation
– Evaluation: Includes scripts such as evaluation.py and evaluation_quality.py, with command-line overrides to resume or set GPU IDs. There’s also support for running via Slurm (evaluation.sh).
– Generation: The generation.py script takes test audio and produces predicted FLAME parameters (.npy files). Further, there is a rendering pipeline (render_param.py, render_vert.py) that converts the predictions into videos via Blender.
- Comparison Branch – Diffusion Model (FaceDiffuser)
– “diffusion” Folder: Contains an alternative model implementation (and related evaluation/generation scripts) for comparing with the VQ-VAE approach.
- External Dependencies / Reused Code
– “deps” Folder: Contains re-implemented and extended code for FLAME, HuBERT, and other borrowed modules. These represent the external libraries or adapted code contributing to the model’s features.
– “static” Folder: Likely for the project website or UI elements that could be referenced during demonstration (CSS, JS, images).

##### Relationships and Interactions
- Data flows from raw datasets (3DMEAD) through preprocessing scripts into the “framework” modules.
- The training pipeline has a clear sequence: data loading (using configuration settings) ⟶ feature extraction (using HuBERT and other modules) ⟶ model forward passes (using different submodules like encoders, decoders, and loss functions) ⟶ checkpoint saving. Note the bifurcation for Stage 1 (prior training) and Stage 2 (prediction training using the prior).
- Evaluation and generation modules use the trained checkpoints to generate outputs; these outputs are processed and rendered.
- The diffusion branch (in its own directory) interacts in a similar way but is designed for a slightly different technical process (it implements a diffusion-based generative model for 3D facial animation).

##### Architectural Patterns and Design Principles
- Modular Design: The project is split into self-contained modules (data, model, tools, evaluation) to facilitate experimentation.
- Hydra-based Configuration: The use of Hydra allows dynamic configuration overrides, making the system flexible for different training or inference scenarios.
- Two-Stage Training Pipeline: Clear separation between learning a motion prior and conditioning on speech/emotion for final predictions.
- Extensibility: With separate directories for external dependencies and alternative approaches (diffusion/facediffuser), the architecture is built to be extended or swapped out.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data & Preprocessing
    subgraph "Data & Preprocessing"
        a1["Datasets"]:::data
        a2["3DMEAD Preprocess"]:::data
        a3["Preprocessing Scripts"]:::data
        a1 --> a2
        a2 --> a3
    end

    %% Configuration Management
    subgraph "Configuration Management"
        b1["Configs (Hydra/YAML)"]:::config
    end

    %% Core Framework
    subgraph "Core Framework"
        c1["Core Framework (Modules)"]:::core
    end

    %% Model Training Pipeline
    subgraph "Model Training Pipeline"
        %% Stage 1: Motion Prior Training
        subgraph "Stage 1: Motion Prior Training"
            d1["Prior Script"]:::train
            d2["Prior Model"]:::train
            d3["Prior Loss"]:::train
            d4["Prior Config"]:::train
            d1 --> d2
            d2 --> d3
            d3 --> d4
        end
        %% Stage 2: Prediction Training
        subgraph "Stage 2: Prediction Training"
            e1["Predict Script"]:::train
            e2["Predict Model"]:::train
            e3["Predict Loss"]:::train
            e4["Predict Config"]:::train
            e5["Main Training"]:::train
            e1 --> e2
            e2 --> e3
            e3 --> e4
            e4 --> e5
        end
    end

    %% Evaluation & Generation
    subgraph "Evaluation & Generation"
        f1["Evaluation Script"]:::eval
        f2["Quality Evaluation"]:::eval
        f3["Eval Shell"]:::eval
        f4["Generation Script"]:::eval
        f5["Render Param"]:::eval
        f6["Render Vert"]:::eval
        f7["Post Processing"]:::eval
        f1 --> f2
        f2 --> f3
        f3 --> f4
        f4 --> f5
        f5 --> f6
        f6 --> f7
    end

    %% Comparison Branch – Diffusion Model
    subgraph "Comparison: Diffusion Model"
        g1["Diffusion Folder"]:::diff
        g2["Diffusion Train"]:::diff
        g3["Diffusion Evaluation"]:::diff
        g1 --> g2
        g2 --> g3
    end

    %% External Dependencies
    subgraph "External Dependencies"
        h1["Dependencies"]:::deps
    end

    %% Static / UI
    subgraph "Static / UI"
        i1["UI Elements"]:::static
    end

    %% Data Flow Connections
    a3 -->|"feeds"| c1
    b1 -->|"configures"| c1
    c1 -->|"initiates"| d1
    d4 -->|"checkpoint"| e1
    e5 -->|"produces"| f1
    c1 -->|"alternatively initiates"| g1
    h1 -->|"supports"| c1
    f7 -->|"renders to"| i1

    %% Click Events for Data & Preprocessing
    click a1 "https://github.com/uuembodiedsocialai/probtalk3d/tree/main/datasets"
    click a2 "https://github.com/uuembodiedsocialai/probtalk3d/tree/main/datasets/3DMEAD_preprocess"
    click a3 "https://github.com/uuembodiedsocialai/probtalk3d/blob/main/pre_process/param_to_vert.py"

    %% Click Event for Configuration Management
    click b1 "https://github.com/uuembodiedsocialai/probtalk3d/tree/main/configs"

    %% Click Event for Core Framework
    click c1 "https://github.com/uuembodiedsocialai/probtalk3d/tree/main/framework"

    %% Click Events for Stage 1: Motion Prior Training
    click d1 "https://github.com/uuembodiedsocialai/probtalk3d/blob/main/train_vqvae_prior.sh"
    click d2 "https://github.com/uuembodiedsocialai/probtalk3d/blob/main/framework/model/model_vqvae_prior.py"
    click d3 "https://github.com/uuembodiedsocialai/probtalk3d/blob/main/framework/model/losses/compute_vqvae_prior.py"
    click d4 "https://github.com/uuembodiedsocialai/probtalk3d/blob/main/configs/model/model_vqvae_prior.yaml"

    %% Click Events for Stage 2: Prediction Training
    click e1 "https://github.com/uuembodiedsocialai/probtalk3d/blob/main/train_vqvae_pred.sh"
    click e2 "https://github.com/uuembodiedsocialai/probtalk3d/blob/main/framework/model/model_vqvae_pred.py"
    click e3 "https://github.com/uuembodiedsocialai/probtalk3d/blob/main/framework/model/losses/compute_vqvae_pred.py"
    click e4 "https://github.com/uuembodiedsocialai/probtalk3d/blob/main/configs/model/model_vqvae_pred.yaml"
    click e5 "https://github.com/uuembodiedsocialai/probtalk3d/blob/main/train_all.py"

    %% Click Events for Evaluation & Generation
    click f1 "https://github.com/uuembodiedsocialai/probtalk3d/blob/main/evaluation.py"
    click f2 "https://github.com/uuembodiedsocialai/probtalk3d/blob/main/evaluation_quality.py"
    click f3 "https://github.com/uuembodiedsocialai/probtalk3d/blob/main/evaluation.sh"
    click f4 "https://github.com/uuembodiedsocialai/probtalk3d/blob/main/generation.py"
    click f5 "https://github.com/uuembodiedsocialai/probtalk3d/blob/main/render_param.py"
    click f6 "https://github.com/uuembodiedsocialai/probtalk3d/blob/main/render_vert.py"
    click f7 "https://github.com/uuembodiedsocialai/probtalk3d/tree/main/post_process"

    %% Click Events for Comparison: Diffusion Model
    click g1 "https://github.com/uuembodiedsocialai/probtalk3d/tree/main/diffusion"
    click g2 "https://github.com/uuembodiedsocialai/probtalk3d/blob/main/diffusion/train_facediff.sh"
    click g3 "https://github.com/uuembodiedsocialai/probtalk3d/blob/main/diffusion/evaluation_facediff.py"

    %% Click Event for External Dependencies
    click h1 "https://github.com/uuembodiedsocialai/probtalk3d/tree/main/deps"

    %% Click Event for Static / UI
    click i1 "https://github.com/uuembodiedsocialai/probtalk3d/tree/main/static"

    %% Styles
    classDef data fill:#ffd1dc,stroke:#cc0077,stroke-width:2px;
    classDef config fill:#cce5ff,stroke:#004085,stroke-width:2px;
    classDef core fill:#d4edda,stroke:#155724,stroke-width:2px;
    classDef train fill:#fff3cd,stroke:#856404,stroke-width:2px;
    classDef eval fill:#e2e3e5,stroke:#383d41,stroke-width:2px;
    classDef diff fill:#f8d7da,stroke:#721c24,stroke-width:2px;
    classDef deps fill:#d1ecf1,stroke:#0c5460,stroke-width:2px;
    classDef static fill:#fceabb,stroke:#8a6d3b,stroke-width:2px;
```
