# tlemangen/freetalkdiff architecture

> Architecture diagram of the GitHub repository tlemangen/freetalkdiff, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/tlemangen/freetalkdiff
- Repository: https://github.com/tlemangen/freetalkdiff

## Overview

FreeTalkDiff generates a talking-face video from an identity video and a motion-driving video. The inference script reads and aligns both videos, runs the generation pipeline, enhances generated faces, composites them back into the original frames, and writes the result. The pipeline combines identity and structure embeddings with inpainting diffusion, then applies temporal noise reduction. Masking and face parsing support alignment/recovery; other documented components are shown where their role is supported. Some pipeline internals are unsampled, so uncertain call relationships are omitted.

Main entry points: [`scripts/inference.py`](https://github.com/tlemangen/freetalkdiff/blob/main/scripts/inference.py) (Inference runner), [`pipelines/freetalkdiff.py`](https://github.com/tlemangen/freetalkdiff/blob/main/pipelines/freetalkdiff.py) (FreeTalkDiff), [`modules/face_alignment.py`](https://github.com/tlemangen/freetalkdiff/blob/main/modules/face_alignment.py) (Face alignment), [`pipelines/sd15_inpaint_ipadapter_faceid_pipeline.py`](https://github.com/tlemangen/freetalkdiff/blob/main/pipelines/sd15_inpaint_ipadapter_faceid_pipeline.py) (Inpaint IP-Adapter)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_inference["Inference workflow"]
  node_inference_script["Inference runner<br/>[inference.py]"]
  node_config["Inference config<br/>[inference.yaml]"]
  node_identity_video["Identity video"]
  node_driven_video["Driving video"]
  node_output_video["Generated video"]
end

subgraph group_generation["Generation pipeline"]
  node_freetalk["FreeTalkDiff<br/>[freetalkdiff.py]"]
  node_diffusion["Inpaint IP-Adapter"]
  node_structurist["Structurist<br/>[structurist.py]"]
  node_controller["Structure controller"]
  node_noise_sensor["Noise sensor<br/>[noise_sensor.py]"]
end

subgraph group_modules["Face processing"]
  node_alignment["Face alignment<br/>[face_alignment.py]"]
  node_masker["Face masker<br/>[masker.py]"]
  node_segmenter["MediaPipe segmenter"]
  node_enhancer["Face enhancer<br/>[face_enhancer.py]"]
end

subgraph group_external["Models and media"]
  node_model_hub[("Hugging Face models")]
  node_video_io["Video I/O"]
end

node_user(("Video user"))

node_user -->|"runs"| node_inference_script
node_inference_script -->|"loads"| node_config
node_video_io -->|"reads videos"| node_inference_script
node_identity_video -->|"supplies identity"| node_inference_script
node_driven_video -->|"supplies motion"| node_inference_script
node_inference_script -->|"aligns and crops"| node_alignment
node_alignment -->|"masks recovery"| node_masker
node_masker -->|"segments frames"| node_segmenter
node_inference_script -->|"generates"| node_freetalk
node_freetalk -->|"inherits and invokes"| node_diffusion
node_freetalk -->|"prepares structure"| node_structurist
node_freetalk -->|"refines embeddings"| node_controller
node_freetalk -->|"reduces flicker"| node_noise_sensor
node_diffusion -.->|"loads pretrained models"| node_model_hub
node_inference_script -->|"enhances faces"| node_enhancer
node_inference_script -->|"recovers frames"| node_alignment
node_inference_script -->|"writes video"| node_video_io
node_inference_script -->|"produces"| node_output_video

click node_inference_script "https://github.com/tlemangen/freetalkdiff/blob/main/scripts/inference.py"
click node_config "https://github.com/tlemangen/freetalkdiff/blob/main/configs/inference.yaml"
click node_alignment "https://github.com/tlemangen/freetalkdiff/blob/main/modules/face_alignment.py"
click node_masker "https://github.com/tlemangen/freetalkdiff/blob/main/modules/masker.py"
click node_segmenter "https://github.com/tlemangen/freetalkdiff/blob/main/modules/mediapipe_segmenter.py"
click node_freetalk "https://github.com/tlemangen/freetalkdiff/blob/main/pipelines/freetalkdiff.py"
click node_diffusion "https://github.com/tlemangen/freetalkdiff/blob/main/pipelines/sd15_inpaint_ipadapter_faceid_pipeline.py"
click node_structurist "https://github.com/tlemangen/freetalkdiff/blob/main/modules/structurist.py"
click node_controller "https://github.com/tlemangen/freetalkdiff/blob/main/modules/structure_controller.py"
click node_noise_sensor "https://github.com/tlemangen/freetalkdiff/blob/main/modules/noise_sensor.py"
click node_enhancer "https://github.com/tlemangen/freetalkdiff/blob/main/modules/face_enhancer.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_inference_script,node_config,node_identity_video,node_driven_video,node_output_video,node_user toneBlue
class node_freetalk,node_diffusion,node_structurist,node_controller,node_noise_sensor toneAmber
class node_alignment,node_masker,node_segmenter,node_enhancer toneMint
class node_model_hub,node_video_io toneRose
```

## Components

### Inference workflow

- [Inference runner](https://github.com/tlemangen/freetalkdiff/blob/main/scripts/inference.py): `scripts/inference.py`
- [Inference config](https://github.com/tlemangen/freetalkdiff/blob/main/configs/inference.yaml): `configs/inference.yaml`
- **Identity video**
- **Driving video**
- **Generated video**

### Generation pipeline

- [FreeTalkDiff](https://github.com/tlemangen/freetalkdiff/blob/main/pipelines/freetalkdiff.py): `pipelines/freetalkdiff.py`
- [Inpaint IP-Adapter](https://github.com/tlemangen/freetalkdiff/blob/main/pipelines/sd15_inpaint_ipadapter_faceid_pipeline.py): `pipelines/sd15_inpaint_ipadapter_faceid_pipeline.py`
- [Structurist](https://github.com/tlemangen/freetalkdiff/blob/main/modules/structurist.py): `modules/structurist.py`
- [Structure controller](https://github.com/tlemangen/freetalkdiff/blob/main/modules/structure_controller.py): `modules/structure_controller.py`
- [Noise sensor](https://github.com/tlemangen/freetalkdiff/blob/main/modules/noise_sensor.py): `modules/noise_sensor.py`

### Face processing

- [Face alignment](https://github.com/tlemangen/freetalkdiff/blob/main/modules/face_alignment.py): `modules/face_alignment.py`
- [Face masker](https://github.com/tlemangen/freetalkdiff/blob/main/modules/masker.py): `modules/masker.py`
- [MediaPipe segmenter](https://github.com/tlemangen/freetalkdiff/blob/main/modules/mediapipe_segmenter.py): `modules/mediapipe_segmenter.py`
- [Face enhancer](https://github.com/tlemangen/freetalkdiff/blob/main/modules/face_enhancer.py): `modules/face_enhancer.py`

### Models and media

- **Hugging Face models**
- **Video I/O**

### Other components

- **Video user**

## Connections

- Video user → Inference runner: runs
- Inference runner → Inference config: loads
- Video I/O → Inference runner: reads videos
- Identity video → Inference runner: supplies identity
- Driving video → Inference runner: supplies motion
- Inference runner → Face alignment: aligns and crops
- Face alignment → Face masker: masks recovery
- Face masker → MediaPipe segmenter: segments frames
- Inference runner → FreeTalkDiff: generates
- FreeTalkDiff → Inpaint IP-Adapter: inherits and invokes
- FreeTalkDiff → Structurist: prepares structure
- FreeTalkDiff → Structure controller: refines embeddings
- FreeTalkDiff → Noise sensor: reduces flicker
- Inpaint IP-Adapter → Hugging Face models: loads pretrained models
- Inference runner → Face enhancer: enhances faces
- Inference runner → Face alignment: recovers frames
- Inference runner → Video I/O: writes video
- Inference runner → Generated video: produces