# sourangshupal/betterdb-yt-collab architecture

> Architecture diagram of the GitHub repository sourangshupal/betterdb-yt-collab, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-05). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/sourangshupal/betterdb-yt-collab
- Repository: https://github.com/sourangshupal/betterdb-yt-collab

## Overview

This repository is a FastAPI RAG demo that creates real Redis data for BetterDB observability. The user workflow uploads a PDF for extraction, chunking, embedding and storage, then submits queries that are rate-limited, embedded, checked against a semantic cache, matched to stored chunks, answered by an LLM, and recorded in cache and session keys. Stats and health endpoints expose Redis key counts and configuration. BetterDB monitoring and MCP workflows are README-documented external capabilities; their implementation is not in this repository.

Main entry points: [`rag/pipeline.py`](https://github.com/sourangshupal/betterdb-yt-collab/blob/master/rag/pipeline.py) (Query orchestration), [`rag/main.py`](https://github.com/sourangshupal/betterdb-yt-collab/blob/master/rag/main.py) (FastAPI endpoints), [`rag/config.py`](https://github.com/sourangshupal/betterdb-yt-collab/blob/master/rag/config.py) (Client configuration)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_api["API"]
  node_api["FastAPI endpoints<br/>[main.py]"]
  node_stats["Health and stats<br/>[main.py]"]
end

subgraph group_rag["RAG pipeline"]
  node_pdf["PDF ingestion<br/>[pipeline.py]"]
  node_chunks["Chunking and embeddings<br/>[pipeline.py]"]
  node_query["Query orchestration<br/>[pipeline.py]"]
  node_limiter["Rate limiting<br/>[pipeline.py]"]
  node_cachelookup["Semantic cache lookup<br/>[pipeline.py]"]
  node_retrieval["Document retrieval<br/>[pipeline.py]"]
  node_generation["Answer generation<br/>[pipeline.py]"]
  node_cachewrite["Cache and session writes<br/>[pipeline.py]"]
end

subgraph group_infra["Redis and AI services"]
  node_redis[("Redis / Valkey")]
  node_openai["OpenAI API"]
  node_config["Client configuration<br/>[config.py]"]
end

subgraph group_ops["Observability"]
  node_betterdb["BetterDB monitoring"]
  node_mcp["BetterDB MCP"]
end

node_user(("API user"))
node_claude(("Claude Code user"))

node_user -->|"uploads or queries"| node_api
node_api -->|"routes PDF"| node_pdf
node_api -->|"routes query"| node_query
node_api -->|"serves status"| node_stats
node_pdf -->|"extracts and embeds"| node_chunks
node_chunks -->|"requests embeddings"| node_openai
node_pdf -->|"writes document hashes"| node_redis
node_query -->|"checks limits"| node_limiter
node_query -->|"embeds query"| node_openai
node_query -->|"checks cache"| node_cachelookup
node_cachelookup -->|"scans cache keys"| node_redis
node_query -->|"retrieves on miss"| node_retrieval
node_retrieval -->|"scans document keys"| node_redis
node_query -->|"requests answer"| node_generation
node_generation -->|"calls chat completion"| node_openai
node_query -->|"stores answer and session"| node_cachewrite
node_cachewrite -->|"writes cache and session"| node_redis
node_limiter -->|"increments counters"| node_redis
node_api -->|"scans health and stats"| node_redis
node_config -->|"creates client"| node_redis
node_config -->|"creates client"| node_openai
node_redis -.->|"observed by agent"| node_betterdb
node_betterdb -->|"serves monitoring data"| node_mcp
node_claude -->|"asks questions"| node_mcp

click node_api "https://github.com/sourangshupal/betterdb-yt-collab/blob/master/rag/main.py"
click node_pdf "https://github.com/sourangshupal/betterdb-yt-collab/blob/master/rag/pipeline.py"
click node_chunks "https://github.com/sourangshupal/betterdb-yt-collab/blob/master/rag/pipeline.py"
click node_query "https://github.com/sourangshupal/betterdb-yt-collab/blob/master/rag/pipeline.py"
click node_limiter "https://github.com/sourangshupal/betterdb-yt-collab/blob/master/rag/pipeline.py"
click node_cachelookup "https://github.com/sourangshupal/betterdb-yt-collab/blob/master/rag/pipeline.py"
click node_retrieval "https://github.com/sourangshupal/betterdb-yt-collab/blob/master/rag/pipeline.py"
click node_generation "https://github.com/sourangshupal/betterdb-yt-collab/blob/master/rag/pipeline.py"
click node_cachewrite "https://github.com/sourangshupal/betterdb-yt-collab/blob/master/rag/pipeline.py"
click node_config "https://github.com/sourangshupal/betterdb-yt-collab/blob/master/rag/config.py"
click node_stats "https://github.com/sourangshupal/betterdb-yt-collab/blob/master/rag/main.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_api,node_stats,node_user,node_claude toneBlue
class node_pdf,node_chunks,node_query,node_limiter,node_cachelookup,node_retrieval,node_generation,node_cachewrite toneAmber
class node_redis,node_openai,node_config toneMint
class node_betterdb,node_mcp toneRose
```

## Components

### API

- [FastAPI endpoints](https://github.com/sourangshupal/betterdb-yt-collab/blob/master/rag/main.py): `rag/main.py`
- [Health and stats](https://github.com/sourangshupal/betterdb-yt-collab/blob/master/rag/main.py): `rag/main.py`

### RAG pipeline

- [PDF ingestion](https://github.com/sourangshupal/betterdb-yt-collab/blob/master/rag/pipeline.py): `rag/pipeline.py`
- [Chunking and embeddings](https://github.com/sourangshupal/betterdb-yt-collab/blob/master/rag/pipeline.py): `rag/pipeline.py`
- [Query orchestration](https://github.com/sourangshupal/betterdb-yt-collab/blob/master/rag/pipeline.py): `rag/pipeline.py`
- [Rate limiting](https://github.com/sourangshupal/betterdb-yt-collab/blob/master/rag/pipeline.py): `rag/pipeline.py`
- [Semantic cache lookup](https://github.com/sourangshupal/betterdb-yt-collab/blob/master/rag/pipeline.py): `rag/pipeline.py`
- [Document retrieval](https://github.com/sourangshupal/betterdb-yt-collab/blob/master/rag/pipeline.py): `rag/pipeline.py`
- [Answer generation](https://github.com/sourangshupal/betterdb-yt-collab/blob/master/rag/pipeline.py): `rag/pipeline.py`
- [Cache and session writes](https://github.com/sourangshupal/betterdb-yt-collab/blob/master/rag/pipeline.py): `rag/pipeline.py`

### Redis and AI services

- **Redis / Valkey**
- **OpenAI API**
- [Client configuration](https://github.com/sourangshupal/betterdb-yt-collab/blob/master/rag/config.py): `rag/config.py`

### Observability

- **BetterDB monitoring**
- **BetterDB MCP**

### Other components

- **API user**
- **Claude Code user**

## Connections

- API user → FastAPI endpoints: uploads or queries (evidence: `README.md`)
- FastAPI endpoints → PDF ingestion: routes PDF (evidence: `rag/main.py`)
- FastAPI endpoints → Query orchestration: routes query (evidence: `rag/main.py`)
- FastAPI endpoints → Health and stats: serves status (evidence: `rag/main.py`)
- PDF ingestion → Chunking and embeddings: extracts and embeds (evidence: `rag/pipeline.py`)
- Chunking and embeddings → OpenAI API: requests embeddings (evidence: `rag/pipeline.py`)
- PDF ingestion → Redis / Valkey: writes document hashes (evidence: `rag/pipeline.py`)
- Query orchestration → Rate limiting: checks limits (evidence: `rag/pipeline.py`)
- Query orchestration → OpenAI API: embeds query (evidence: `rag/pipeline.py`)
- Query orchestration → Semantic cache lookup: checks cache (evidence: `rag/pipeline.py`)
- Semantic cache lookup → Redis / Valkey: scans cache keys (evidence: `rag/pipeline.py`)
- Query orchestration → Document retrieval: retrieves on miss (evidence: `rag/pipeline.py`)
- Document retrieval → Redis / Valkey: scans document keys (evidence: `rag/pipeline.py`)
- Query orchestration → Answer generation: requests answer (evidence: `rag/pipeline.py`)
- Answer generation → OpenAI API: calls chat completion (evidence: `rag/pipeline.py`)
- Query orchestration → Cache and session writes: stores answer and session (evidence: `rag/pipeline.py`)
- Cache and session writes → Redis / Valkey: writes cache and session (evidence: `rag/pipeline.py`)
- Rate limiting → Redis / Valkey: increments counters (evidence: `rag/pipeline.py`)
- FastAPI endpoints → Redis / Valkey: scans health and stats (evidence: `rag/main.py`)
- Client configuration → Redis / Valkey: creates client (evidence: `rag/config.py`)
- Client configuration → OpenAI API: creates client (evidence: `rag/config.py`)
- Redis / Valkey → BetterDB monitoring: observed by agent (evidence: `README.md`)
- BetterDB monitoring → BetterDB MCP: serves monitoring data (evidence: `README.md`)
- Claude Code user → BetterDB MCP: asks questions (evidence: `README.md`)
