# kaist-ami/multitalk architecture

> Architecture diagram of the GitHub repository kaist-ami/multitalk, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-02-04). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/kaist-ami/multitalk
- Repository: https://github.com/kaist-ami/multitalk

## Overview

##### Project Type and Purpose
- This is a PyTorch-based machine-learning pipeline for generating 3D talking-head animations across multiple languages.
- It implements a two-stage approach: Stage 1 learns a discrete motion prior (via VQ-VAE), and Stage 2 drives synthesis from speech features (facebook/wav2vec2).
- It includes modules for data processing, model training, inference/demo, and quantitative evaluation (LVE and AVLR).

##### File-Structure Analysis
Top-level directories and their roles:
- MultiTalk_dataset: Raw multilingual video/audio + annotations + download & pre-process scripts.
- sample_dataset: Small subset of data for quick testing/demo.
- base/: Framework-agnostic abstractions (baseTrainer, base_model, config loader, utility functions).
- config/: YAML configuration for each training stage (stage1.yaml, stage2.yaml).
- dataset/: DataLoader implementations assembling audio + mesh sequences per language.
- models/:
- lib/: Core building blocks (quantizer, wav2vec wrapper, base_models).
- stage1_vocaset.py & stage2.py: Stage-specific model definitions.
- metrics/: Loss functions.
- eval_avlr/: Audio-Visual Lip Reading evaluation code.
- main/: Python entry points for training, testing, metric calculation, demo orchestration.
- scripts/: Bash wrappers (demo.sh, train_multi.sh, test.sh).
- utils/: Miscellaneous helper functions (model utilities, general utilities).

##### README Insights
- Two pre-trained checkpoints required (stage1.pth.tar, stage2.pth.tar) plus a FLAME topology template.
- Demo and evaluation workflows illustrated via shell scripts.
- External deps: PyTorch, transformers (wav2vec2), psbody-mesh, ffmpeg.

1. Data Ingestion & Pre-processing
- Raw videos/audio (MultiTalk_dataset → download_and_process.py → .npy meshes + wavs)
2. DataLoader Layer
- dataset/data_loader_multi.py
3. Stage 1: Discrete Motion Prior
- base/baseTrainer.py → models/lib/quantizer.py → models/stage1_vocaset.py
- Outputs codebooks & discrete motion tokens
4. Stage 2: Speech-Driven Motion Synthesis
- wav2vec2 feature extractor (transformers) → models/stage2.py
- Consumes tokens + audio features → Mesh parameter predictions
5. Mesh Reconstruction & Rendering
- psbody-mesh + FLAME_sample.ply
6. Evaluation
- Lip Vertex Error (main/cal_metric.py)
- AVLR (eval_avlr/eval_avlr.py → muavic checkpoints)
7. CLI / Orchestration
- scripts/*.sh and main/*.py drivers
8. Configuration
- config/multi/*.yaml

- MultiTalk_dataset → DataLoader → Stage 1 Trainer → Stage1 Checkpoint & Codebook
- wav2vec2 Model → Stage 2 Trainer (pulls Stage1 codebook) → Stage2 Checkpoint
- Inference Pipeline (demo.sh) loads Stage1 & Stage2 → generates mesh sequences
- Output meshes + wavs → Evaluation modules (LVE & AVLR) → metrics

c. Architectural Patterns & Principles:
- Modular layering (Data → Model → Training → Inference → Evaluation)
- Two-stage pipeline (decoupled prior learning vs. synthesis)
- Config-driven behavior (YAML files steer training hyperparameters)
- Reusable abstractions (baseTrainer, base_model)

d. Technologies & Frameworks:
- PyTorch 1.12.0, transformers (wav2vec2-xlsr-53)
- psbody-mesh (FLAME)
- ffmpeg (video/audio processing)
- Python scripts + Bash wrappers

## Diagram (Mermaid)

```mermaid
flowchart TD
  %% Data Layer
  subgraph "Data Layer"
    DataIngestion["Data Ingestion & Pre-processing"]:::data
    DataLoader["DataLoader Layer"]:::data
  end

  %% Stage 1 Training
  subgraph "Stage 1: Discrete Motion Prior"
    S1Trainer["Stage 1 Trainer"]:::model
    S1Quantizer["VQ-VAE Quantizer"]:::model
    S1Wrapper["Stage1 Model Wrapper"]:::model
  end

  %% Stage 2 Training
  subgraph "Stage 2: Speech-Driven Motion Synthesis"
    S2Trainer["Stage 2 Trainer"]:::model
    Wav2vecWrapper["Wav2vec2 Wrapper"]:::model
    S2Model["Stage2 Model Definition"]:::model
  end

  %% Configuration & Utilities
  subgraph "Config & Utilities"
    ConfigLoader["Configuration Loader"]:::utility
    Stage1Config["stage1.yaml"]:::config
    Stage2Config["stage2.yaml"]:::config
    DemoConfig["demo.yaml"]:::config
    Loss["Loss Functions"]:::utility
    BaseUtil["base/utilities.py"]:::utility
    GenUtil["utils/util.py"]:::utility
  end

  %% Orchestration & CLI
  subgraph "Orchestration & CLI"
    EntryTrain1["main/train_multi_vq.py"]:::cli
    EntryTrain2["main/test_multi_pred.py"]:::cli
    EntryDemo["main/demo_dir.py"]:::cli
    ScriptsTrain["train_multi.sh"]:::cli
    ScriptsTest["test.sh"]:::cli
    ScriptsDemo["demo.sh"]:::cli
  end

  %% Evaluation
  subgraph "Evaluation"
    LVE["Lip Vertex Error"]:::eval
    AVLR["Audio-Visual Lip Reading"]:::eval
  end

  %% External Dependencies
  subgraph "External Dependencies"
    Wav2vec["pretrained wav2vec2"]:::external
    Psbody["psbody-mesh & FLAME"]:::external
    Ffmpeg["ffmpeg"]:::external
  end

  %% Data flows
  DataIngestion --> DataLoader
  DataLoader --> S1Trainer
  S1Trainer --> S1Quantizer
  S1Quantizer --> S1Wrapper
  S1Wrapper -->|quantized tokens| S2Trainer
  Wav2vec --> Wav2vecWrapper
  Wav2vecWrapper --> S2Model
  S2Trainer --> S2Model
  S2Model -->|mesh params| Psbody
  Psbody --> EntryDemo
  Ffmpeg --> DataIngestion

  %% Config & control flows
  ConfigLoader -.-> Stage1Config
  ConfigLoader -.-> Stage2Config
  ConfigLoader -.-> DemoConfig
  Stage1Config -.-> S1Trainer
  Stage2Config -.-> S2Trainer
  DemoConfig -.-> EntryDemo

  %% Utilities and loss
  Loss --> S1Trainer
  Loss --> S2Trainer
  BaseUtil --> S1Trainer
  BaseUtil --> S2Trainer
  GenUtil --> S1Trainer
  GenUtil --> S2Trainer

  %% CLI orchestration
  ScriptsTrain --> EntryTrain1
  ScriptsTest --> EntryTrain2
  ScriptsDemo --> EntryDemo
  EntryTrain1 --> S1Trainer
  EntryTrain2 --> S2Trainer
  EntryDemo --> S2Model

  %% Evaluation flows
  S2Model --> LVE
  S2Model --> AVLR

  %% Legend
  classDef data fill:#e0f7fa,stroke:#006064
  classDef model fill:#e8f5e9,stroke:#1b5e20
  classDef eval fill:#fff3e0,stroke:#e65100
  classDef external fill:#eeeeee,stroke:#424242
  classDef config fill:#f3e5f5,stroke:#6a1b9a
  classDef utility fill:#fbe9e7,stroke:#bf360c
  classDef cli fill:#fce4ec,stroke:#880e4f

  %% Click Events
  click DataIngestion "https://github.com/kaist-ami/multitalk/blob/main/MultiTalk_dataset/download_and_process.py"
  click DataLoader "https://github.com/kaist-ami/multitalk/blob/main/dataset/data_loader_multi.py"
  click S1Trainer "https://github.com/kaist-ami/multitalk/blob/main/base/baseTrainer.py"
  click S1Quantizer "https://github.com/kaist-ami/multitalk/blob/main/models/lib/quantizer.py"
  click S1Wrapper "https://github.com/kaist-ami/multitalk/blob/main/models/stage1_vocaset.py"
  click S2Trainer "https://github.com/kaist-ami/multitalk/blob/main/main/train_multi_pred.py"
  click Wav2vecWrapper "https://github.com/kaist-ami/multitalk/blob/main/models/lib/wav2vec.py"
  click S2Model "https://github.com/kaist-ami/multitalk/blob/main/models/stage2.py"
  click ConfigLoader "https://github.com/kaist-ami/multitalk/blob/main/base/config.py"
  click Stage1Config "https://github.com/kaist-ami/multitalk/blob/main/config/multi/stage1.yaml"
  click Stage2Config "https://github.com/kaist-ami/multitalk/blob/main/config/multi/stage2.yaml"
  click DemoConfig "https://github.com/kaist-ami/multitalk/blob/main/config/multi/demo.yaml"
  click Loss "https://github.com/kaist-ami/multitalk/blob/main/metrics/loss.py"
  click LVE "https://github.com/kaist-ami/multitalk/blob/main/main/cal_metric.py"
  click AVLR "https://github.com/kaist-ami/multitalk/blob/main/eval_avlr/eval_avlr.py"
  click ScriptsTrain "https://github.com/kaist-ami/multitalk/blob/main/scripts/train_multi.sh"
  click ScriptsTest "https://github.com/kaist-ami/multitalk/blob/main/scripts/test.sh"
  click ScriptsDemo "https://github.com/kaist-ami/multitalk/blob/main/scripts/demo.sh"
  click EntryTrain1 "https://github.com/kaist-ami/multitalk/blob/main/main/train_multi_vq.py"
  click EntryTrain2 "https://github.com/kaist-ami/multitalk/blob/main/main/test_multi_pred.py"
  click EntryDemo "https://github.com/kaist-ami/multitalk/blob/main/main/demo_dir.py"
  click BaseUtil "https://github.com/kaist-ami/multitalk/blob/main/base/utilities.py"
  click GenUtil "https://github.com/kaist-ami/multitalk/blob/main/utils/util.py"
```
