How siddarthaa/whisperx works
Project Type and Purpose
• WhisperX is a full-stack, single-page Python application (using Streamlit via App.py) for media processing. It ingests audio/video, transcribes to text, creates embeddings, indexes them, and supports three key features:
• Summarization
• RAG-based Question Answering
• Quiz Generation
High-Level File Structure Analysis
• Top‐level
• App.py – the Streamlit frontend and main orchestrator
• QuizGen.py – standalone quiz generation module
• Setup.py / requirements.txt – environment and dependency setup
• Prompts/ – contains prompt templates (instructions, question types, summarization, quizzes)
• No separately deployed microservices; the design is a modular, pipeline-style monolith.
Main Architectural Components
a. User Interface (Streamlit)
• Upload widget, parameter settings, result display
b. Transcription Service
• OpenAI Whisper or Tiny.en invoked via Python bindings
c. Embedding Service
• All-MPNet-Base-V2 model generates vector embeddings
d. Vector Store / Retrieval (FAISS)
• In‐memory or on‐disk index for semantic search
e. Summarization Module
• Llama3 / Gemini via Ollama for condensing transcripts
f. RAG Question Answering Module
• Query FAISS, retrieve relevant chunks, feed to LLM for answer generation
g. Quiz Generation Module
• QuizGen.py + Prompts/Quiz templates to build MCQs, fill-in, true/false, short answer
h. Prompt Repository
• Prompts/ folder containing all prompt templates and instructions
Component Relationships & Data Flow
1. User uploads media in the UI
→ App.py sends file to Transcription Service
→ Transcript data flows to Embedding Service
→ Embeddings are stored/indexed in FAISS
2. Summarization Request
→ Transcript → LLM Summarization (Llama3/Gemini) → UI
3. Q&A Request
→ User question in UI → FAISS retrieval of top-k chunks → LLM RAG pipeline → UI
4. Quiz Generation
→ Summarized text + quiz templates → QuizGen.py → Generated quizzes → UI
Architectural Patterns & Principles
• Modular Pipeline: clear separation of concerns (transcription, embedding, retrieval, generation)
• RAG (Retrieval-Augmented Generation) pattern for Q&A
• Template-driven prompt management for extensibility
• Single-process orchestration (no heavy microservices)