How astorfi/lip-reading-deeplearning works
This repository implements coupled 3D-CNN audio-visual matching, with lip-reading as its demonstrated application. The workflow prepares mouth and speech feature inputs, feeds modality-specific networks through a factory, and trains/evaluates their matching representation. Lip tracking and speech-feature preparation are user-facing input tools; the README says users supply the overall input pipeline. The source excerpts show the tracker and speech transforms, while some training and evaluation details are documented or indexed but not fully shown.
Main entry points: train.py Training Workflow · nets_factory.py Network Factory · VisualizeLip.py Lip Tracker · input_feature.py Audio Dataset
Components 18
Input Preparation
- Input Video
- Speech Audio
- Audio Dataset
code/speech-input/input_feature.py - Speech Features
code/speech-input/input_feature.py - Feature Transforms
code/speech-input/input_feature.py - Paired Features
Coupled Models
- Network Factory
code/training_evaluation/nets/nets_factory.py - Mouth Network
code/training_evaluation/nets/lipread_mouth.py - Speech Network
code/training_evaluation/nets/lipread_speech.py
Training and Evaluation
- Training Workflow
code/training_evaluation/train.py - Contrastive Loss
code/training_evaluation/auxiliary/losses.py - ROC Metrics
code/training_evaluation/roc_curve/calculate_roc.py - Metric Plots
code/training_evaluation/roc_curve/PlotROC.py - Matching Result
Lip Tracking
- Lip Tracker
code/lip_tracking/VisualizeLip.py - Mouth Crops
code/lip_tracking/VisualizeLip.py - Annotated Video
code/lip_tracking/VisualizeLip.py
Other components
- User
Connections 18
- User → Input Video: provides
README.rst - User → Speech Audio: provides
README.rst - Input Video → Lip Tracker: processes
README.rst - Lip Tracker → Mouth Crops: saves crops
code/lip_tracking/VisualizeLip.py - Lip Tracker → Annotated Video: writes video
code/lip_tracking/VisualizeLip.py - Speech Audio → Audio Dataset: loads audio
code/speech-input/input_feature.py - Audio Dataset → Speech Features: extracts features
code/speech-input/input_feature.py - Speech Features → Feature Transforms: transforms
code/speech-input/input_feature.py - Mouth Crops → Paired Features: forms visual input
README.rst - Feature Transforms → Paired Features: forms speech input
README.rst - Paired Features → Training Workflow: provides batches
README.rst - Training Workflow → Network Factory: selects networks
code/training_evaluation/train.py - Network Factory → Mouth Network: dispatches
code/training_evaluation/nets/nets_factory.py - Network Factory → Speech Network: dispatches
code/training_evaluation/nets/nets_factory.py - Training Workflow → Contrastive Loss: computes objective
code/training_evaluation/train.py - Training Workflow → ROC Metrics: calculates metrics
code/training_evaluation/train.py - Training Workflow → Matching Result: learns correspondence
README.rst - Training Workflow → Metric Plots: may visualize metrics