# astorfi/lip-reading-deeplearning architecture

> Architecture diagram of the GitHub repository astorfi/lip-reading-deeplearning, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-02). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/astorfi/lip-reading-deeplearning
- Repository: https://github.com/astorfi/lip-reading-deeplearning

## Overview

This repository implements coupled 3D-CNN audio-visual matching, with lip-reading as its demonstrated application. The workflow prepares mouth and speech feature inputs, feeds modality-specific networks through a factory, and trains/evaluates their matching representation. Lip tracking and speech-feature preparation are user-facing input tools; the README says users supply the overall input pipeline. The source excerpts show the tracker and speech transforms, while some training and evaluation details are documented or indexed but not fully shown.

Main entry points: [`code/training_evaluation/train.py`](https://github.com/astorfi/lip-reading-deeplearning/blob/master/code/training_evaluation/train.py) (Training Workflow), [`code/training_evaluation/nets/nets_factory.py`](https://github.com/astorfi/lip-reading-deeplearning/blob/master/code/training_evaluation/nets/nets_factory.py) (Network Factory), [`code/lip_tracking/VisualizeLip.py`](https://github.com/astorfi/lip-reading-deeplearning/blob/master/code/lip_tracking/VisualizeLip.py) (Lip Tracker), [`code/speech-input/input_feature.py`](https://github.com/astorfi/lip-reading-deeplearning/blob/master/code/speech-input/input_feature.py) (Audio Dataset)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_input["Input Preparation"]
  node_video["Input Video"]
  node_audio["Speech Audio"]
  node_speech_dataset["Audio Dataset<br/>[input_feature.py]"]
  node_speech_features["Speech Features<br/>[input_feature.py]"]
  node_feature_transforms["Feature Transforms<br/>[input_feature.py]"]
  node_paired_inputs["Paired Features"]
end

subgraph group_model["Coupled Models"]
  node_network_factory["Network Factory<br/>[nets_factory.py]"]
  node_mouth_network["Mouth Network<br/>[lipread_mouth.py]"]
  node_speech_network["Speech Network<br/>[lipread_speech.py]"]
end

subgraph group_learning["Training and Evaluation"]
  node_trainer["Training Workflow<br/>[train.py]"]
  node_contrastive_loss["Contrastive Loss<br/>[losses.py]"]
  node_roc_metrics["ROC Metrics<br/>[calculate_roc.py]"]
  node_plots["Metric Plots<br/>[PlotROC.py]"]
  node_matching_result["Matching Result"]
end

subgraph group_visualization["Lip Tracking"]
  node_lip_tracker["Lip Tracker<br/>[VisualizeLip.py]"]
  node_mouth_crops["Mouth Crops<br/>[VisualizeLip.py]"]
  node_annotated_video["Annotated Video<br/>[VisualizeLip.py]"]
end

node_user(("User"))

node_user -->|"provides"| node_video
node_user -->|"provides"| node_audio
node_video -->|"processes"| node_lip_tracker
node_lip_tracker -->|"saves crops"| node_mouth_crops
node_lip_tracker -->|"writes video"| node_annotated_video
node_audio -->|"loads audio"| node_speech_dataset
node_speech_dataset -->|"extracts features"| node_speech_features
node_speech_features -->|"transforms"| node_feature_transforms
node_mouth_crops -->|"forms visual input"| node_paired_inputs
node_feature_transforms -->|"forms speech input"| node_paired_inputs
node_paired_inputs -->|"provides batches"| node_trainer
node_trainer -->|"selects networks"| node_network_factory
node_network_factory -->|"dispatches"| node_mouth_network
node_network_factory -->|"dispatches"| node_speech_network
node_trainer -->|"computes objective"| node_contrastive_loss
node_trainer -->|"calculates metrics"| node_roc_metrics
node_trainer -->|"learns correspondence"| node_matching_result
node_trainer -.->|"may visualize metrics"| node_plots

click node_lip_tracker "https://github.com/astorfi/lip-reading-deeplearning/blob/master/code/lip_tracking/VisualizeLip.py"
click node_mouth_crops "https://github.com/astorfi/lip-reading-deeplearning/blob/master/code/lip_tracking/VisualizeLip.py"
click node_annotated_video "https://github.com/astorfi/lip-reading-deeplearning/blob/master/code/lip_tracking/VisualizeLip.py"
click node_speech_dataset "https://github.com/astorfi/lip-reading-deeplearning/blob/master/code/speech-input/input_feature.py"
click node_speech_features "https://github.com/astorfi/lip-reading-deeplearning/blob/master/code/speech-input/input_feature.py"
click node_feature_transforms "https://github.com/astorfi/lip-reading-deeplearning/blob/master/code/speech-input/input_feature.py"
click node_trainer "https://github.com/astorfi/lip-reading-deeplearning/blob/master/code/training_evaluation/train.py"
click node_network_factory "https://github.com/astorfi/lip-reading-deeplearning/blob/master/code/training_evaluation/nets/nets_factory.py"
click node_mouth_network "https://github.com/astorfi/lip-reading-deeplearning/blob/master/code/training_evaluation/nets/lipread_mouth.py"
click node_speech_network "https://github.com/astorfi/lip-reading-deeplearning/blob/master/code/training_evaluation/nets/lipread_speech.py"
click node_contrastive_loss "https://github.com/astorfi/lip-reading-deeplearning/blob/master/code/training_evaluation/auxiliary/losses.py"
click node_roc_metrics "https://github.com/astorfi/lip-reading-deeplearning/blob/master/code/training_evaluation/roc_curve/calculate_roc.py"
click node_plots "https://github.com/astorfi/lip-reading-deeplearning/blob/master/code/training_evaluation/roc_curve/PlotROC.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_video,node_audio,node_speech_dataset,node_speech_features,node_feature_transforms,node_paired_inputs,node_user toneBlue
class node_network_factory,node_mouth_network,node_speech_network toneAmber
class node_trainer,node_contrastive_loss,node_roc_metrics,node_plots,node_matching_result toneMint
class node_lip_tracker,node_mouth_crops,node_annotated_video toneRose
```

## Components

### Input Preparation

- **Input Video**
- **Speech Audio**
- [Audio Dataset](https://github.com/astorfi/lip-reading-deeplearning/blob/master/code/speech-input/input_feature.py): `code/speech-input/input_feature.py`
- [Speech Features](https://github.com/astorfi/lip-reading-deeplearning/blob/master/code/speech-input/input_feature.py): `code/speech-input/input_feature.py`
- [Feature Transforms](https://github.com/astorfi/lip-reading-deeplearning/blob/master/code/speech-input/input_feature.py): `code/speech-input/input_feature.py`
- **Paired Features**

### Coupled Models

- [Network Factory](https://github.com/astorfi/lip-reading-deeplearning/blob/master/code/training_evaluation/nets/nets_factory.py): `code/training_evaluation/nets/nets_factory.py`
- [Mouth Network](https://github.com/astorfi/lip-reading-deeplearning/blob/master/code/training_evaluation/nets/lipread_mouth.py): `code/training_evaluation/nets/lipread_mouth.py`
- [Speech Network](https://github.com/astorfi/lip-reading-deeplearning/blob/master/code/training_evaluation/nets/lipread_speech.py): `code/training_evaluation/nets/lipread_speech.py`

### Training and Evaluation

- [Training Workflow](https://github.com/astorfi/lip-reading-deeplearning/blob/master/code/training_evaluation/train.py): `code/training_evaluation/train.py`
- [Contrastive Loss](https://github.com/astorfi/lip-reading-deeplearning/blob/master/code/training_evaluation/auxiliary/losses.py): `code/training_evaluation/auxiliary/losses.py`
- [ROC Metrics](https://github.com/astorfi/lip-reading-deeplearning/blob/master/code/training_evaluation/roc_curve/calculate_roc.py): `code/training_evaluation/roc_curve/calculate_roc.py`
- [Metric Plots](https://github.com/astorfi/lip-reading-deeplearning/blob/master/code/training_evaluation/roc_curve/PlotROC.py): `code/training_evaluation/roc_curve/PlotROC.py`
- **Matching Result**

### Lip Tracking

- [Lip Tracker](https://github.com/astorfi/lip-reading-deeplearning/blob/master/code/lip_tracking/VisualizeLip.py): `code/lip_tracking/VisualizeLip.py`
- [Mouth Crops](https://github.com/astorfi/lip-reading-deeplearning/blob/master/code/lip_tracking/VisualizeLip.py): `code/lip_tracking/VisualizeLip.py`
- [Annotated Video](https://github.com/astorfi/lip-reading-deeplearning/blob/master/code/lip_tracking/VisualizeLip.py): `code/lip_tracking/VisualizeLip.py`

### Other components

- **User**

## Connections

- User → Input Video: provides (evidence: `README.rst`)
- User → Speech Audio: provides (evidence: `README.rst`)
- Input Video → Lip Tracker: processes (evidence: `README.rst`)
- Lip Tracker → Mouth Crops: saves crops (evidence: `code/lip_tracking/VisualizeLip.py`)
- Lip Tracker → Annotated Video: writes video (evidence: `code/lip_tracking/VisualizeLip.py`)
- Speech Audio → Audio Dataset: loads audio (evidence: `code/speech-input/input_feature.py`)
- Audio Dataset → Speech Features: extracts features (evidence: `code/speech-input/input_feature.py`)
- Speech Features → Feature Transforms: transforms (evidence: `code/speech-input/input_feature.py`)
- Mouth Crops → Paired Features: forms visual input (evidence: `README.rst`)
- Feature Transforms → Paired Features: forms speech input (evidence: `README.rst`)
- Paired Features → Training Workflow: provides batches (evidence: `README.rst`)
- Training Workflow → Network Factory: selects networks (evidence: `code/training_evaluation/train.py`)
- Network Factory → Mouth Network: dispatches (evidence: `code/training_evaluation/nets/nets_factory.py`)
- Network Factory → Speech Network: dispatches (evidence: `code/training_evaluation/nets/nets_factory.py`)
- Training Workflow → Contrastive Loss: computes objective (evidence: `code/training_evaluation/train.py`)
- Training Workflow → ROC Metrics: calculates metrics (evidence: `code/training_evaluation/train.py`)
- Training Workflow → Matching Result: learns correspondence (evidence: `README.rst`)
- Training Workflow → Metric Plots: may visualize metrics
