How farazkhanai/avsr_project works
This is a Flask-based audio-visual speech-recognition app. A user uploads a video in the web UI; Flask saves it temporarily and calls the fusion engine, which runs Whisper transcription and lip-motion scoring before returning a sentence and confidence tokens for display. The README also describes upload limits, cleanup, and feature-stream diagnostics; the sampled client code confirms sentence and token display, but not diagnostics. The graph follows the sampled route and pipeline calls; unsampled template details are not inferred.
Main entry points: app.py Flask Routes · fusion_engine.py Late Fusion · audio_pipeline.py Audio Transcription · main.js Upload and Results UI
Components 13
Web Interface
- Upload and Results UI
static/js/main.js - AVSR Page
templates/index.html - UI Styling
static/css/style.css
Request Handling
- Flask Routes
app.py - Temporary Uploads
AVSR Inference
- Late Fusion
utils/fusion_engine.py - Audio Transcription
utils/audio_pipeline.py - Lip Motion Scoring
utils/visual_pipeline.py
Model and Media
- Whisper Model
- FFmpeg
- OpenCV Video Reader
Other components
- User
- Browser
Connections 19
- User → Browser: uses
- Browser → Upload and Results UI: loads
- Flask Routes → AVSR Page: renders
- Upload and Results UI → Flask Routes: posts video
- Flask Routes → Temporary Uploads: saves video
- Flask Routes → Late Fusion: calls prediction
- Late Fusion → Audio Transcription: transcribes video
- Late Fusion → Lip Motion Scoring: scores motion
- Audio Transcription → FFmpeg: extracts audio
- Audio Transcription → Whisper Model: transcribes speech
- Lip Motion Scoring → OpenCV Video Reader: reads frames
- Audio Transcription → Late Fusion: returns transcript
- Lip Motion Scoring → Late Fusion: returns score
- Late Fusion → Flask Routes: returns prediction
- Flask Routes → Temporary Uploads: deletes video
- Flask Routes → Upload and Results UI: returns JSON
- Upload and Results UI → Browser: displays results
- Browser → User: shows output
- Browser → UI Styling: loads styles