How tlemangen/freetalkdiff works
FreeTalkDiff generates a talking-face video from an identity video and a motion-driving video. The inference script reads and aligns both videos, runs the generation pipeline, enhances generated faces, composites them back into the original frames, and writes the result. The pipeline combines identity and structure embeddings with inpainting diffusion, then applies temporal noise reduction. Masking and face parsing support alignment/recovery; other documented components are shown where their role is supported. Some pipeline internals are unsampled, so uncertain call relationships are omitted.
Main entry points: inference.py Inference runner · freetalkdiff.py FreeTalkDiff · face_alignment.py Face alignment · sd15_inpaint_ipadapter_faceid_pipeline.py Inpaint IP-Adapter
Components 17
Inference workflow
- Inference runner
scripts/inference.py - Inference config
configs/inference.yaml - Identity video
- Driving video
- Generated video
Generation pipeline
- FreeTalkDiff
pipelines/freetalkdiff.py - Inpaint IP-Adapter
pipelines/sd15_inpaint_ipadapter_faceid_pipeline.py - Structurist
modules/structurist.py - Structure controller
modules/structure_controller.py - Noise sensor
modules/noise_sensor.py
Face processing
- Face alignment
modules/face_alignment.py - Face masker
modules/masker.py - MediaPipe segmenter
modules/mediapipe_segmenter.py - Face enhancer
modules/face_enhancer.py
Models and media
- Hugging Face models
- Video I/O
Other components
- Video user
Connections 18
- Video user → Inference runner: runs
- Inference runner → Inference config: loads
- Video I/O → Inference runner: reads videos
- Identity video → Inference runner: supplies identity
- Driving video → Inference runner: supplies motion
- Inference runner → Face alignment: aligns and crops
- Face alignment → Face masker: masks recovery
- Face masker → MediaPipe segmenter: segments frames
- Inference runner → FreeTalkDiff: generates
- FreeTalkDiff → Inpaint IP-Adapter: inherits and invokes
- FreeTalkDiff → Structurist: prepares structure
- FreeTalkDiff → Structure controller: refines embeddings
- FreeTalkDiff → Noise sensor: reduces flicker
- Inpaint IP-Adapter → Hugging Face models: loads pretrained models
- Inference runner → Face enhancer: enhances faces
- Inference runner → Face alignment: recovers frames
- Inference runner → Video I/O: writes video
- Inference runner → Generated video: produces