# antgroup/echomimic architecture

> Architecture diagram of the GitHub repository antgroup/echomimic, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-13). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/antgroup/echomimic
- Repository: https://github.com/antgroup/echomimic

## Overview

##### Project type and purpose
- EchoMimic is a research-grade Python library and toolkit for generating lifelike, audio- and/or landmark-driven portrait animations using diffusion models plus a transformer-based motion module.
- It provides command-line inference scripts (e.g., infer_audio2vid.py), a demo script for motion synchronization (demo_motion_sync.py), and a Gradio-based Web GUI (webgui.py).

##### File structure
- Top-level directories:
- assets: test inputs (images, audio, videos)
- configs: YAML files for inference and prompt settings
- src: core code, subdivided into models, pipelines, utils
- Key modules:
- src/models: UNet variants (2D, 3D, echo), transformer modules, motion_module, face_locator, Whisper audio processor
- src/pipelines: orchestrate the overall inference workflow for different modes (audio only, pose only, combined)
- src/utils: helper functions for image I/O, drawing, face landmarks, motion computation
- Inference scripts at top level invoke pipelines, load configs, read pretrained weights, write videos.
- The Web GUI (webgui.py) wraps inference scripts in a Gradio server.

##### README for additional insights
- Describes Audio-Driven, Pose-Driven, and Audio+Pose-Driven modes
- Points out 10× acceleration variants (infer_*_acc.py)
- Dependencies: diffusion checkpoints, Whisper Tiny, ffmpeg for video assembly
- Inference configurable via YAML under configs/prompts and configs/inference

##### Components
1. Input Layer
- Reference image(s)
- Driver audio file (and/or landmark sequence)
2. Preprocessing
- Whisper audio2feature module (audio → feature vectors)
- Face landmark detection (MediaPipe or face_landmarker)
- Optional: pose loading or drawing function
3. Pipeline Orchestrator (src/pipelines)
- Context builder: packages audio features + landmarks + prompts
- Mode-specific pipelines (pipeline_echo_mimic, pipeline_echo_mimic_pose, etc.)
4. Core Model Stack
- Motion Module (transformer_2d or transformer_3d)
- Diffusion UNet(s): denoising_unet, reference_unet, unet_3d_echo
- Attention modules, self-attention, mutual attention
- Face locator network
5. Postprocessing
- Frame decoding → images → assemble into video (ffmpeg)
- Draw_utils for landmarks overlay (debug/demo)
6. Output Layer
- Generated video file
7. Configuration & Checkpoints
- YAML config files (prompts + inference settings)
- Pretrained model weights directory
8. Web GUI Layer
- Gradio server (webgui.py) exposing above pipeline as REST/HTML UI

b. Relationships and interactions
- Inputs feed Preprocessing → Pipeline Orchestrator
- Orchestrator loads Config → instantiates Core Model Stack → yields raw frames
- Postprocessing consumes frames → outputs final video
- Web GUI invokes same Orchestrator via CLI calls or direct imports

c. Architectural patterns & design principles
- Modular pipeline pattern: separate preprocessing, core model, postprocessing
- Strategy pattern: different pipeline classes for audio-only, pose-only, combined
- Configuration-driven: decode YAML at runtime to change prompts, test cases
- Reusable utility modules for image/audio/landmark I/O

d. Technologies, frameworks, libraries
- PyTorch for model definitions & inference
- OpenAI Whisper for audio feature extraction
- Diffusers style UNet implementations
- MediaPipe or TFLite for face landmark detection
- ffmpeg for assembling video
- Gradio for Web UI

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Input Layer
    subgraph "Input Layer"
        imgs["Reference Images"]:::data
        aud["Driver Audio Files"]:::data
        pose_seq["Landmark Sequences"]:::data
    end

    %% Preprocessing
    subgraph "Preprocessing"
        whisper["Whisper Audio→Feature"]:::model
        face_core["face_landmark.py"]:::external
        mp_utils["mp_utils.py"]:::external
        tflite["blaze_face_short_range.tflite"]:::external
        fl_model["face_landmarker_v2_with_blendshapes.task"]:::external
        img_utils["img_utils.py"]:::external
    end

    %% Pipeline Orchestrator
    subgraph "Pipeline Orchestrator"
        ctx["Context Builder"]:::config
        pipe_audio["pipeline_echo_mimic.py"]:::orchestrator
        pipe_audio_acc["pipeline_echo_mimic_acc.py"]:::orchestrator
        pipe_pose["pipeline_echo_mimic_pose.py"]:::orchestrator
        pipe_pose_acc["pipeline_echo_mimic_pose_acc.py"]:::orchestrator
        utils_pipe["utils.py"]:::orchestrator
    end

    %% Core Model Stack
    subgraph "Core Model Stack"
        motion_mod["motion_module.py"]:::model
        trans2d["transformer_2d.py"]:::model
        trans3d["transformer_3d.py"]:::model
        unet2d_cond["unet_2d_condition.py"]:::model
        unet2d_blk["unet_2d_blocks.py"]:::model
        unet3d["unet_3d.py"]:::model
        unet3d_blk["unet_3d_blocks.py"]:::model
        unet3d_echo["unet_3d_echo.py"]:::model
        attention["attention.py"]:::model
        msa["mutual_self_attention.py"]:::model
        face_loc["face_locator.py"]:::model
        resnet["resnet.py"]:::model
    end

    %% Postprocessing
    subgraph "Postprocessing"
        draw["draw_utils.py"]:::external
        motion_utils["motion_utils.py"]:::external
        step_func["step_func.py"]:::external
        util["util.py"]:::external
    end

    %% Configs & Checkpoints
    subgraph "Configs & Checkpoints"
        inf_v1["inference_v1.yaml"]:::config
        inf_v2["inference_v2.yaml"]:::config
        prom_anim["animation.yaml"]:::config
        prom_anim_acc["animation_acc.yaml"]:::config
        prom_pose["animation_pose.yaml"]:::config
        prom_pose_acc["animation_pose_acc.yaml"]:::config
    end

    %% Inference Scripts & Demo
    subgraph "Inference Scripts & Demo"
        inf_audio["infer_audio2vid.py"]:::external
        inf_audio_acc["infer_audio2vid_acc.py"]:::external
        inf_pose["infer_audio2vid_pose.py"]:::external
        inf_pose_acc["infer_audio2vid_pose_acc.py"]:::external
        demo["demo_motion_sync.py"]:::external
    end

    %% Web GUI Layer
    web["webgui.py"]:::external

    %% Output Layer
    subgraph "Output Layer"
        vid["Generated Video File"]:::data
    end

    %% Connections
    aud -->|"audio→features"| whisper
    imgs -->|"image→landmarks"| face_core
    pose_seq -->|landmarks| face_core
    mp_utils --> face_core
    tflite --> face_core
    fl_model --> face_core
    img_utils --> face_core

    whisper --> ctx
    face_core --> ctx

    ctx -->|"build context"| pipe_audio
    ctx --> pipe_audio_acc
    ctx -->|"optional pose"| pipe_pose
    ctx --> pipe_pose_acc
    utils_pipe --> pipe_audio
    utils_pipe --> pipe_pose

    pipe_audio --> motion_mod
    pipe_audio_acc --> motion_mod
    pipe_pose --> motion_mod
    pipe_pose_acc --> motion_mod

    motion_mod --> trans2d
    motion_mod --> trans3d
    trans2d --> unet2d_cond
    trans2d --> unet2d_blk
    trans3d --> unet3d
    trans3d --> unet3d_blk
    unet3d_blk --> unet3d_echo

    attention --> trans2d
    attention --> trans3d
    msa --> trans2d
    msa --> trans3d
    face_loc --> unet2d_cond
    face_loc --> unet3d

    unet2d_cond --> draw
    unet2d_blk --> draw
    unet3d_echo --> draw
    draw --> motion_utils
    motion_utils --> step_func
    step_func --> util
    util --> vid

    inf_audio --> ctx
    inf_audio_acc --> ctx
    inf_pose --> ctx
    inf_pose_acc --> ctx
    demo --> ctx

    web --> inf_audio
    web --> inf_pose
    web --> inf_audio_acc
    web --> inf_pose_acc
    web --> demo

    inf_v1 --> ctx
    inf_v2 --> ctx
    prom_anim --> ctx
    prom_anim_acc --> ctx
    prom_pose --> ctx
    prom_pose_acc --> ctx

    %% Click Events
    click imgs "https://github.com/antgroup/echomimic/tree/main/assets/test_imgs"
    click aud "https://github.com/antgroup/echomimic/tree/main/assets/test_audios"
    click pose_seq "https://github.com/antgroup/echomimic/blob/main/assets/test_pose_demo/*.pkl"
    click whisper "https://github.com/antgroup/echomimic/blob/main/src/models/whisper/audio2feature.py"
    click face_core "https://github.com/antgroup/echomimic/blob/main/src/utils/face_landmark.py"
    click mp_utils "https://github.com/antgroup/echomimic/blob/main/src/utils/mp_utils.py"
    click tflite "https://github.com/antgroup/echomimic/blob/main/src/utils/mp_models/blaze_face_short_range.tflite"
    click fl_model "https://github.com/antgroup/echomimic/blob/main/src/utils/mp_models/face_landmarker_v2_with_blendshapes.task"
    click img_utils "https://github.com/antgroup/echomimic/blob/main/src/utils/img_utils.py"
    click ctx "https://github.com/antgroup/echomimic/blob/main/src/pipelines/context.py"
    click pipe_audio "https://github.com/antgroup/echomimic/blob/main/src/pipelines/pipeline_echo_mimic.py"
    click pipe_audio_acc "https://github.com/antgroup/echomimic/blob/main/src/pipelines/pipeline_echo_mimic_acc.py"
    click pipe_pose "https://github.com/antgroup/echomimic/blob/main/src/pipelines/pipeline_echo_mimic_pose.py"
    click pipe_pose_acc "https://github.com/antgroup/echomimic/blob/main/src/pipelines/pipeline_echo_mimic_pose_acc.py"
    click utils_pipe "https://github.com/antgroup/echomimic/blob/main/src/pipelines/utils.py"
    click motion_mod "https://github.com/antgroup/echomimic/blob/main/src/models/motion_module.py"
    click trans2d "https://github.com/antgroup/echomimic/blob/main/src/models/transformer_2d.py"
    click trans3d "https://github.com/antgroup/echomimic/blob/main/src/models/transformer_3d.py"
    click unet2d_cond "https://github.com/antgroup/echomimic/blob/main/src/models/unet_2d_condition.py"
    click unet2d_blk "https://github.com/antgroup/echomimic/blob/main/src/models/unet_2d_blocks.py"
    click unet3d "https://github.com/antgroup/echomimic/blob/main/src/models/unet_3d.py"
    click unet3d_blk "https://github.com/antgroup/echomimic/blob/main/src/models/unet_3d_blocks.py"
    click unet3d_echo "https://github.com/antgroup/echomimic/blob/main/src/models/unet_3d_echo.py"
    click attention "https://github.com/antgroup/echomimic/blob/main/src/models/attention.py"
    click msa "https://github.com/antgroup/echomimic/blob/main/src/models/mutual_self_attention.py"
    click face_loc "https://github.com/antgroup/echomimic/blob/main/src/models/face_locator.py"
    click resnet "https://github.com/antgroup/echomimic/blob/main/src/models/resnet.py"
    click draw "https://github.com/antgroup/echomimic/blob/main/src/utils/draw_utils.py"
    click motion_utils "https://github.com/antgroup/echomimic/blob/main/src/utils/motion_utils.py"
    click step_func "https://github.com/antgroup/echomimic/blob/main/src/utils/step_func.py"
    click util "https://github.com/antgroup/echomimic/blob/main/src/utils/util.py"
    click inf_v1 "https://github.com/antgroup/echomimic/blob/main/configs/inference/inference_v1.yaml"
    click inf_v2 "https://github.com/antgroup/echomimic/blob/main/configs/inference/inference_v2.yaml"
    click prom_anim "https://github.com/antgroup/echomimic/blob/main/configs/prompts/animation.yaml"
    click prom_anim_acc "https://github.com/antgroup/echomimic/blob/main/configs/prompts/animation_acc.yaml"
    click prom_pose "https://github.com/antgroup/echomimic/blob/main/configs/prompts/animation_pose.yaml"
    click prom_pose_acc "https://github.com/antgroup/echomimic/blob/main/configs/prompts/animation_pose_acc.yaml"
    click inf_audio "https://github.com/antgroup/echomimic/blob/main/infer_audio2vid.py"
    click inf_audio_acc "https://github.com/antgroup/echomimic/blob/main/infer_audio2vid_acc.py"
    click inf_pose "https://github.com/antgroup/echomimic/blob/main/infer_audio2vid_pose.py"
    click inf_pose_acc "https://github.com/antgroup/echomimic/blob/main/infer_audio2vid_pose_acc.py"
    click demo "https://github.com/antgroup/echomimic/blob/main/demo_motion_sync.py"
    click web "https://github.com/antgroup/echomimic/blob/main/webgui.py"

    %% Styles
    classDef data fill:#D0E6FF,stroke:#333,stroke-width:1px
    classDef model fill:#D0FFD6,stroke:#333,stroke-width:1px
    classDef config fill:#FFF8B0,stroke:#333,stroke-width:1px
    classDef external fill:#FFE2B0,stroke:#333,stroke-width:1px
    classDef orchestrator fill:#E8D7FF,stroke:#333,stroke-width:1px
```
