# kumar2302github/rag-powered-chatbot-for-youtube-transcript-understanding_ architecture

> Architecture diagram of the GitHub repository kumar2302github/rag-powered-chatbot-for-youtube-transcript-understanding_, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-13). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/kumar2302github/rag-powered-chatbot-for-youtube-transcript-understanding_
- Repository: https://github.com/kumar2302github/rag-powered-chatbot-for-youtube-transcript-understanding_

## Overview

##### Project Type and Purpose
- This is a full-stack, modular RAG (Retrieval-Augmented Generation) system in a single Python application, exposed via a Streamlit web UI.
- It fetches YouTube transcripts, chunks and embeds them, stores vectors locally, performs hybrid retrieval, and generates conversational answers using an LLM (e.g., Google Gemini).

##### High-Level File Structure Analysis
- Top-level:
- `app.py` (Streamlit entry point)
- `rag.py` (core RAG pipeline: chunking, embedding, retrieval, generation)
- `transcription.py` (YouTube transcript fetcher)
- `requirements.txt`, `.streamlit/secrets.toml` (config & secrets)
- `Test_rag.ipynb` (experimental notebook)
- No separate “frontend” vs. “backend” folders, but modules logically separate:
- UI layer in `app.py`
- Service layer in `rag.py` and `transcription.py`
- Storage layer using ChromaDB

##### Key Components
a. User (Web Browser)
b. Streamlit Frontend
- Hosts the chat interface, sidebar controls for Top-K, threshold, chunk size/overlap, MMR λ, reset/delete
c. Transcript Service
- `transcription.py` calls YouTube Transcript API
d. Chunking & Embedding Service
- Sentence-Transformers model
e. Vector Store (ChromaDB)
- Local persistent vector database, one collection per video
f. Retrieval Engine
- Score threshold + HDBSCAN clustering + MMR re-ranking
g. LLM API
- External (Gemini) or pluggable endpoint
h. Notebook (Test_rag.ipynb)—optional, for development/testing

##### Data Flow & Interactions
1. User submits a video URL or query via Streamlit UI
2. Frontend invokes Transcript Service → retrieves raw transcript
3. Transcript passed to Chunking & Embedding → generates embeddings
4. Embeddings upserted into ChromaDB
5. On user query:
a. Query embedded via same embedding model
b. Query embedding sent to Vector Store → retrieves candidate chunks
c. Apply threshold filtering → HDBSCAN → MMR → final chunk set
6. Selected chunks + original query → sent to LLM API
7. LLM returns answer → Frontend displays response back to user

##### Architectural Patterns & Design Principles
- Modular separation: UI, data ingestion, embedding, retrieval, generation
- Pipeline pattern: sequential processing stages (extract → embed → store → retrieve → generate)
- Hybrid retrieval: combining relevance (distance) and diversity (MMR + clustering)
- Pluggable components: embedding model, vector DB, LLM endpoint

##### Final Checklist
- [ ] Components: User, Streamlit UI, Transcription, Embedding, Vector DB, Retrieval, LLM, Notebook
- [ ] Annotations on pipeline stages (chunk size, overlap, MMR λ, thresholds)

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% User Interaction
    User["User (Web Browser)"]:::external
    User -->|"interacts with"| UI

    %% Frontend Layer
    subgraph Frontend Layer
        UI["Streamlit UI"]:::ui
    end

    %% Backend Layer
    subgraph Backend Layer
        TS["Transcript Service"]:::service
        ES["Chunking & Embedding Service"]:::service
        VS["Vector Store (ChromaDB)"]:::storage
        RE["Retrieval Engine"]:::service
        LI["LLM API Interface"]:::service
    end

    %% External Services
    subgraph External Services
        YT["YouTube API"]:::external
        LLM["LLM API (Gemini)"]:::external
        Notebook["Dev/Test Notebook"]:::external
        Config["Configuration/Secrets"]:::external
        Req["Dependency Manifest"]:::external
    end

    %% Data Flow
    UI -->|"submit URL/query"| TS
    TS -->|"fetch transcript (JSON)"| YT
    TS -->|"raw transcript"| ES
    ES -->|"send embeddings"| VS
    UI -->|"query text"| ES
    ES -->|"query embedding"| VS
    VS -->|"candidate chunks"| RE
    RE -->|"LLM prompt"| LI
    Config -->|"API key/config"| LI
    LI -->|"generate answer"| LLM
    LLM -->|"answer text"| LI
    LI -->|"return answer"| UI
    UI -->|"displays answer"| User

    %% Optional Development
    Notebook --- ES

    %% Click Events
    click UI "https://github.com/kumar2302github/rag-powered-chatbot-for-youtube-transcript-understanding_/blob/main/app.py"
    click TS "https://github.com/kumar2302github/rag-powered-chatbot-for-youtube-transcript-understanding_/blob/main/transcription.py"
    click ES "https://github.com/kumar2302github/rag-powered-chatbot-for-youtube-transcript-understanding_/blob/main/rag.py"
    click VS "https://github.com/kumar2302github/rag-powered-chatbot-for-youtube-transcript-understanding_/blob/main/rag.py"
    click RE "https://github.com/kumar2302github/rag-powered-chatbot-for-youtube-transcript-understanding_/blob/main/rag.py"
    click LI "https://github.com/kumar2302github/rag-powered-chatbot-for-youtube-transcript-understanding_/blob/main/rag.py"
    click Notebook "https://github.com/kumar2302github/rag-powered-chatbot-for-youtube-transcript-understanding_/blob/main/Test_rag.ipynb"
    click Config "https://github.com/kumar2302github/rag-powered-chatbot-for-youtube-transcript-understanding_/blob/main/.streamlit/secrets.toml"
    click Req "https://github.com/kumar2302github/rag-powered-chatbot-for-youtube-transcript-understanding_/blob/main/requirements.txt"

    %% Styling
    classDef ui fill:#ADD8E6,stroke:#333,stroke-width:1px
    classDef service fill:#90EE90,stroke:#333,stroke-width:1px
    classDef storage fill:#FFA500,stroke:#333,stroke-width:1px
    classDef external fill:#D3D3D3,stroke:#333,stroke-width:1px
```
