# mouadbra/rag-pipeline architecture

> Architecture diagram of the GitHub repository mouadbra/rag-pipeline, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-27). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/mouadbra/rag-pipeline
- Repository: https://github.com/mouadbra/rag-pipeline

## Overview

Discord RAG Pipeline ingests server messages, stores their text and embeddings, and answers questions through an LLM-selected RAG or SQL path. The frontend launches scraping and questions, then displays the answer, selected approach, SQL when applicable, and interaction history. The backend entry point owns request orchestration; the scraper and search/query functions perform the domain work. README-described pipeline visualizations are represented by the sampled frontend; unsampled implementation details are not inferred.

Main entry points: [`backend_service/src/modal_app/main.py`](https://github.com/mouadbra/rag-pipeline/blob/main/backend_service/src/modal_app/main.py) (Question endpoint), [`backend_service/src/modal_app/discord.py`](https://github.com/mouadbra/rag-pipeline/blob/main/backend_service/src/modal_app/discord.py) (Channel fetcher), [`frontend_service/src/App.tsx`](https://github.com/mouadbra/rag-pipeline/blob/main/frontend_service/src/App.tsx) (Pipeline UI), [`backend_service/src/modal_app/common.py`](https://github.com/mouadbra/rag-pipeline/blob/main/backend_service/src/modal_app/common.py) (FastAPI runtime)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_frontend["User interface"]
  node_ui["Pipeline UI<br/>[App.tsx]"]
end

subgraph group_api["Backend API"]
  node_api_runtime["FastAPI runtime<br/>[common.py]"]
  node_ask_endpoint["Question endpoint<br/>[main.py]"]
  node_scrape_endpoint["Scrape endpoint<br/>[main.py]"]
  node_volume[("Modal persistent volume<br/>[common.py]")]
end

subgraph group_ingestion["Message ingestion"]
  node_scraper["Guild scraper<br/>[discord.py]"]
  node_channel_fetch["Channel fetcher<br/>[discord.py]"]
  node_database[("Messages and vectors<br/>[main.py]")]
end

subgraph group_query["Question answering"]
  node_decision_tools["Approach tool schema<br/>[common.py]"]
  node_vector_search["RAG similarity search<br/>[main.py]"]
  node_sql_query["SQL execution<br/>[main.py]"]
end

node_user(("User"))
node_discord_api["Discord API"]
node_embed_service["Azure embeddings"]
node_chat_service["Azure chat model"]

node_user -->|"submits inputs"| node_ui
node_ui -->|"POST scrape"| node_scrape_endpoint
node_ui -->|"POST question"| node_ask_endpoint
node_api_runtime -->|"routes requests"| node_scrape_endpoint
node_api_runtime -->|"routes requests"| node_ask_endpoint
node_scrape_endpoint -->|"reloads and commits"| node_volume
node_scrape_endpoint -->|"starts scrape"| node_scraper
node_scraper -->|"fetches channels"| node_discord_api
node_scraper -->|"fetches text channels"| node_channel_fetch
node_channel_fetch -->|"fetches messages"| node_discord_api
node_channel_fetch -->|"creates embeddings"| node_embed_service
node_channel_fetch -->|"writes messages and vectors"| node_database
node_ask_endpoint -->|"requests approach"| node_chat_service
node_ask_endpoint -->|"provides tool schema"| node_decision_tools
node_ask_endpoint -->|"runs RAG branch"| node_vector_search
node_ask_endpoint -->|"runs SQL branch"| node_sql_query
node_vector_search -->|"embeds query"| node_embed_service
node_vector_search -->|"searches vectors and messages"| node_database
node_sql_query -->|"executes query"| node_database
node_ask_endpoint -->|"generates final answer"| node_chat_service
node_database -->|"persists on volume"| node_volume
node_ask_endpoint -->|"returns answer and history"| node_ui

click node_ui "https://github.com/mouadbra/rag-pipeline/blob/main/frontend_service/src/App.tsx"
click node_api_runtime "https://github.com/mouadbra/rag-pipeline/blob/main/backend_service/src/modal_app/common.py"
click node_ask_endpoint "https://github.com/mouadbra/rag-pipeline/blob/main/backend_service/src/modal_app/main.py"
click node_scrape_endpoint "https://github.com/mouadbra/rag-pipeline/blob/main/backend_service/src/modal_app/main.py"
click node_scraper "https://github.com/mouadbra/rag-pipeline/blob/main/backend_service/src/modal_app/discord.py"
click node_channel_fetch "https://github.com/mouadbra/rag-pipeline/blob/main/backend_service/src/modal_app/discord.py"
click node_decision_tools "https://github.com/mouadbra/rag-pipeline/blob/main/backend_service/src/modal_app/common.py"
click node_vector_search "https://github.com/mouadbra/rag-pipeline/blob/main/backend_service/src/modal_app/main.py"
click node_sql_query "https://github.com/mouadbra/rag-pipeline/blob/main/backend_service/src/modal_app/main.py"
click node_database "https://github.com/mouadbra/rag-pipeline/blob/main/backend_service/src/modal_app/main.py"
click node_volume "https://github.com/mouadbra/rag-pipeline/blob/main/backend_service/src/modal_app/common.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_ui,node_user toneBlue
class node_api_runtime,node_ask_endpoint,node_scrape_endpoint,node_volume toneAmber
class node_scraper,node_channel_fetch,node_database,node_discord_api toneMint
class node_decision_tools,node_vector_search,node_sql_query toneRose
class node_embed_service,node_chat_service toneIndigo
```

## Components

### User interface

- [Pipeline UI](https://github.com/mouadbra/rag-pipeline/blob/main/frontend_service/src/App.tsx): `frontend_service/src/App.tsx`

### Backend API

- [FastAPI runtime](https://github.com/mouadbra/rag-pipeline/blob/main/backend_service/src/modal_app/common.py): `backend_service/src/modal_app/common.py`
- [Question endpoint](https://github.com/mouadbra/rag-pipeline/blob/main/backend_service/src/modal_app/main.py): `backend_service/src/modal_app/main.py`
- [Scrape endpoint](https://github.com/mouadbra/rag-pipeline/blob/main/backend_service/src/modal_app/main.py): `backend_service/src/modal_app/main.py`
- [Modal persistent volume](https://github.com/mouadbra/rag-pipeline/blob/main/backend_service/src/modal_app/common.py): `backend_service/src/modal_app/common.py`

### Message ingestion

- [Guild scraper](https://github.com/mouadbra/rag-pipeline/blob/main/backend_service/src/modal_app/discord.py): `backend_service/src/modal_app/discord.py`
- [Channel fetcher](https://github.com/mouadbra/rag-pipeline/blob/main/backend_service/src/modal_app/discord.py): `backend_service/src/modal_app/discord.py`
- [Messages and vectors](https://github.com/mouadbra/rag-pipeline/blob/main/backend_service/src/modal_app/main.py): `backend_service/src/modal_app/main.py`

### Question answering

- [Approach tool schema](https://github.com/mouadbra/rag-pipeline/blob/main/backend_service/src/modal_app/common.py): `backend_service/src/modal_app/common.py`
- [RAG similarity search](https://github.com/mouadbra/rag-pipeline/blob/main/backend_service/src/modal_app/main.py): `backend_service/src/modal_app/main.py`
- [SQL execution](https://github.com/mouadbra/rag-pipeline/blob/main/backend_service/src/modal_app/main.py): `backend_service/src/modal_app/main.py`

### Other components

- **User**
- **Discord API**
- **Azure embeddings**
- **Azure chat model**

## Connections

- User → Pipeline UI: submits inputs
- Pipeline UI → Scrape endpoint: POST scrape
- Pipeline UI → Question endpoint: POST question
- FastAPI runtime → Scrape endpoint: routes requests
- FastAPI runtime → Question endpoint: routes requests
- Scrape endpoint → Modal persistent volume: reloads and commits
- Scrape endpoint → Guild scraper: starts scrape
- Guild scraper → Discord API: fetches channels
- Guild scraper → Channel fetcher: fetches text channels
- Channel fetcher → Discord API: fetches messages
- Channel fetcher → Azure embeddings: creates embeddings
- Channel fetcher → Messages and vectors: writes messages and vectors
- Question endpoint → Azure chat model: requests approach
- Question endpoint → Approach tool schema: provides tool schema
- Question endpoint → RAG similarity search: runs RAG branch
- Question endpoint → SQL execution: runs SQL branch
- RAG similarity search → Azure embeddings: embeds query
- RAG similarity search → Messages and vectors: searches vectors and messages
- SQL execution → Messages and vectors: executes query
- Question endpoint → Azure chat model: generates final answer
- Messages and vectors → Modal persistent volume: persists on volume
- Question endpoint → Pipeline UI: returns answer and history