How kaist-ami/multitalk works
Project Type and Purpose
• This is a PyTorch-based machine-learning pipeline for generating 3D talking-head animations across multiple languages.
• It implements a two-stage approach: Stage 1 learns a discrete motion prior (via VQ-VAE), and Stage 2 drives synthesis from speech features (facebook/wav2vec2).
• It includes modules for data processing, model training, inference/demo, and quantitative evaluation (LVE and AVLR).
File-Structure Analysis
Top-level directories and their roles:
• MultiTalk_dataset: Raw multilingual video/audio + annotations + download & pre-process scripts.
• sample_dataset: Small subset of data for quick testing/demo.
• base/: Framework-agnostic abstractions (baseTrainer, base_model, config loader, utility functions).
• config/: YAML configuration for each training stage (stage1.yaml, stage2.yaml).
• dataset/: DataLoader implementations assembling audio + mesh sequences per language.
• models/:
• lib/: Core building blocks (quantizer, wav2vec wrapper, base_models).
• stage1_vocaset.py & stage2.py: Stage-specific model definitions.
• metrics/: Loss functions.
• eval_avlr/: Audio-Visual Lip Reading evaluation code.
• main/: Python entry points for training, testing, metric calculation, demo orchestration.
• scripts/: Bash wrappers (demo.sh, train_multi.sh, test.sh).
• utils/: Miscellaneous helper functions (model utilities, general utilities).
README Insights
• Two pre-trained checkpoints required (stage1.pth.tar, stage2.pth.tar) plus a FLAME topology template.
• Demo and evaluation workflows illustrated via shell scripts.
• External deps: PyTorch, transformers (wav2vec2), psbody-mesh, ffmpeg.
1. Data Ingestion & Pre-processing
• Raw videos/audio (MultiTalk_dataset → download_and_process.py → .npy meshes + wavs)
2. DataLoader Layer
• dataset/data_loader_multi.py
3. Stage 1: Discrete Motion Prior
• base/baseTrainer.py → models/lib/quantizer.py → models/stage1_vocaset.py
• Outputs codebooks & discrete motion tokens
4. Stage 2: Speech-Driven Motion Synthesis
• wav2vec2 feature extractor (transformers) → models/stage2.py
• Consumes tokens + audio features → Mesh parameter predictions
5. Mesh Reconstruction & Rendering
• psbody-mesh + FLAME_sample.ply
6. Evaluation
• Lip Vertex Error (main/cal_metric.py)
• AVLR (eval_avlr/eval_avlr.py → muavic checkpoints)
7. CLI / Orchestration
• scripts/*.sh and main/*.py drivers
8. Configuration
• config/multi/*.yaml
• MultiTalk_dataset → DataLoader → Stage 1 Trainer → Stage1 Checkpoint & Codebook
• wav2vec2 Model → Stage 2 Trainer (pulls Stage1 codebook) → Stage2 Checkpoint
• Inference Pipeline (demo.sh) loads Stage1 & Stage2 → generates mesh sequences
• Output meshes + wavs → Evaluation modules (LVE & AVLR) → metrics
c. Architectural Patterns & Principles:
• Modular layering (Data → Model → Training → Inference → Evaluation)
• Two-stage pipeline (decoupled prior learning vs. synthesis)
• Config-driven behavior (YAML files steer training hyperparameters)
• Reusable abstractions (baseTrainer, base_model)
d. Technologies & Frameworks:
• PyTorch 1.12.0, transformers (wav2vec2-xlsr-53)
• psbody-mesh (FLAME)
• ffmpeg (video/audio processing)
• Python scripts + Bash wrappers