How mouadbra/rag-pipeline works
Discord RAG Pipeline ingests server messages, stores their text and embeddings, and answers questions through an LLM-selected RAG or SQL path. The frontend launches scraping and questions, then displays the answer, selected approach, SQL when applicable, and interaction history. The backend entry point owns request orchestration; the scraper and search/query functions perform the domain work. README-described pipeline visualizations are represented by the sampled frontend; unsampled implementation details are not inferred.
Main entry points: main.py Question endpoint · discord.py Channel fetcher · App.tsx Pipeline UI · common.py FastAPI runtime
Components 15
User interface
- Pipeline UI
frontend_service/src/App.tsx
Backend API
- FastAPI runtime
backend_service/src/modal_app/common.py - Question endpoint
backend_service/src/modal_app/main.py - Scrape endpoint
backend_service/src/modal_app/main.py - Modal persistent volume
backend_service/src/modal_app/common.py
Message ingestion
- Guild scraper
backend_service/src/modal_app/discord.py - Channel fetcher
backend_service/src/modal_app/discord.py - Messages and vectors
backend_service/src/modal_app/main.py
Question answering
- Approach tool schema
backend_service/src/modal_app/common.py - RAG similarity search
backend_service/src/modal_app/main.py - SQL execution
backend_service/src/modal_app/main.py
Other components
- User
- Discord API
- Azure embeddings
- Azure chat model
Connections 22
- User → Pipeline UI: submits inputs
- Pipeline UI → Scrape endpoint: POST scrape
- Pipeline UI → Question endpoint: POST question
- FastAPI runtime → Scrape endpoint: routes requests
- FastAPI runtime → Question endpoint: routes requests
- Scrape endpoint → Modal persistent volume: reloads and commits
- Scrape endpoint → Guild scraper: starts scrape
- Guild scraper → Discord API: fetches channels
- Guild scraper → Channel fetcher: fetches text channels
- Channel fetcher → Discord API: fetches messages
- Channel fetcher → Azure embeddings: creates embeddings
- Channel fetcher → Messages and vectors: writes messages and vectors
- Question endpoint → Azure chat model: requests approach
- Question endpoint → Approach tool schema: provides tool schema
- Question endpoint → RAG similarity search: runs RAG branch
- Question endpoint → SQL execution: runs SQL branch
- RAG similarity search → Azure embeddings: embeds query
- RAG similarity search → Messages and vectors: searches vectors and messages
- SQL execution → Messages and vectors: executes query
- Question endpoint → Azure chat model: generates final answer
- Messages and vectors → Modal persistent volume: persists on volume
- Question endpoint → Pipeline UI: returns answer and history