# farazkhanai/avsr_project architecture

> Architecture diagram of the GitHub repository farazkhanai/avsr_project, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-27). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/farazkhanai/avsr_project
- Repository: https://github.com/farazkhanai/avsr_project

## Overview

This is a Flask-based audio-visual speech-recognition app. A user uploads a video in the web UI; Flask saves it temporarily and calls the fusion engine, which runs Whisper transcription and lip-motion scoring before returning a sentence and confidence tokens for display. The README also describes upload limits, cleanup, and feature-stream diagnostics; the sampled client code confirms sentence and token display, but not diagnostics. The graph follows the sampled route and pipeline calls; unsampled template details are not inferred.

Main entry points: [`app.py`](https://github.com/farazkhanai/avsr_project/blob/main/app.py) (Flask Routes), [`utils/fusion_engine.py`](https://github.com/farazkhanai/avsr_project/blob/main/utils/fusion_engine.py) (Late Fusion), [`utils/audio_pipeline.py`](https://github.com/farazkhanai/avsr_project/blob/main/utils/audio_pipeline.py) (Audio Transcription), [`static/js/main.js`](https://github.com/farazkhanai/avsr_project/blob/main/static/js/main.js) (Upload and Results UI)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_web["Web Interface"]
  node_client["Upload and Results UI<br/>[main.js]"]
  node_page["AVSR Page<br/>[index.html]"]
  node_styles["UI Styling<br/>[style.css]"]
end

subgraph group_server["Request Handling"]
  node_flask["Flask Routes<br/>[app.py]"]
  node_uploads["Temporary Uploads"]
end

subgraph group_inference["AVSR Inference"]
  node_fusion["Late Fusion<br/>[fusion_engine.py]"]
  node_audio["Audio Transcription<br/>[audio_pipeline.py]"]
  node_visual["Lip Motion Scoring<br/>[visual_pipeline.py]"]
end

subgraph group_integrations["Model and Media"]
  node_whisper["Whisper Model"]
  node_ffmpeg["FFmpeg"]
  node_opencv["OpenCV Video Reader"]
end

node_user(("User"))
node_browser(("Browser"))

node_user -->|"uses"| node_browser
node_browser -->|"loads"| node_client
node_flask -->|"renders"| node_page
node_client -->|"posts video"| node_flask
node_flask -->|"saves video"| node_uploads
node_flask -->|"calls prediction"| node_fusion
node_fusion -->|"transcribes video"| node_audio
node_fusion -->|"scores motion"| node_visual
node_audio -->|"extracts audio"| node_ffmpeg
node_audio -->|"transcribes speech"| node_whisper
node_visual -->|"reads frames"| node_opencv
node_audio -->|"returns transcript"| node_fusion
node_visual -->|"returns score"| node_fusion
node_fusion -->|"returns prediction"| node_flask
node_flask -->|"deletes video"| node_uploads
node_flask -->|"returns JSON"| node_client
node_client -->|"displays results"| node_browser
node_browser -->|"shows output"| node_user
node_browser -->|"loads styles"| node_styles

click node_client "https://github.com/farazkhanai/avsr_project/blob/main/static/js/main.js"
click node_page "https://github.com/farazkhanai/avsr_project/blob/main/templates/index.html"
click node_styles "https://github.com/farazkhanai/avsr_project/blob/main/static/css/style.css"
click node_flask "https://github.com/farazkhanai/avsr_project/blob/main/app.py"
click node_fusion "https://github.com/farazkhanai/avsr_project/blob/main/utils/fusion_engine.py"
click node_audio "https://github.com/farazkhanai/avsr_project/blob/main/utils/audio_pipeline.py"
click node_visual "https://github.com/farazkhanai/avsr_project/blob/main/utils/visual_pipeline.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_client,node_page,node_styles,node_user,node_browser toneBlue
class node_flask,node_uploads toneAmber
class node_fusion,node_audio,node_visual toneMint
class node_whisper,node_ffmpeg,node_opencv toneRose
```

## Components

### Web Interface

- [Upload and Results UI](https://github.com/farazkhanai/avsr_project/blob/main/static/js/main.js): `static/js/main.js`
- [AVSR Page](https://github.com/farazkhanai/avsr_project/blob/main/templates/index.html): `templates/index.html`
- [UI Styling](https://github.com/farazkhanai/avsr_project/blob/main/static/css/style.css): `static/css/style.css`

### Request Handling

- [Flask Routes](https://github.com/farazkhanai/avsr_project/blob/main/app.py): `app.py`
- **Temporary Uploads**

### AVSR Inference

- [Late Fusion](https://github.com/farazkhanai/avsr_project/blob/main/utils/fusion_engine.py): `utils/fusion_engine.py`
- [Audio Transcription](https://github.com/farazkhanai/avsr_project/blob/main/utils/audio_pipeline.py): `utils/audio_pipeline.py`
- [Lip Motion Scoring](https://github.com/farazkhanai/avsr_project/blob/main/utils/visual_pipeline.py): `utils/visual_pipeline.py`

### Model and Media

- **Whisper Model**
- **FFmpeg**
- **OpenCV Video Reader**

### Other components

- **User**
- **Browser**

## Connections

- User → Browser: uses
- Browser → Upload and Results UI: loads
- Flask Routes → AVSR Page: renders
- Upload and Results UI → Flask Routes: posts video
- Flask Routes → Temporary Uploads: saves video
- Flask Routes → Late Fusion: calls prediction
- Late Fusion → Audio Transcription: transcribes video
- Late Fusion → Lip Motion Scoring: scores motion
- Audio Transcription → FFmpeg: extracts audio
- Audio Transcription → Whisper Model: transcribes speech
- Lip Motion Scoring → OpenCV Video Reader: reads frames
- Audio Transcription → Late Fusion: returns transcript
- Lip Motion Scoring → Late Fusion: returns score
- Late Fusion → Flask Routes: returns prediction
- Flask Routes → Temporary Uploads: deletes video
- Flask Routes → Upload and Results UI: returns JSON
- Upload and Results UI → Browser: displays results
- Browser → User: shows output
- Browser → UI Styling: loads styles