# ysn-b13/ask-fsdl-2026 architecture

> Architecture diagram of the GitHub repository ysn-b13/ask-fsdl-2026, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-08-02). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/ysn-b13/ask-fsdl-2026
- Repository: https://github.com/ysn-b13/ask-fsdl-2026

## Overview

#### Purpose and Entry Points
askFSDL is a retrieval-augmented QA system over Full Stack Deep Learning educational content. Users ask questions through a Discord bot or a lightweight Gradio UI; the backend retrieves corpus passages and uses an OpenAI-backed LangChain flow to generate answers.

Primary runtime entry points:
- `app.py` — application backend / Gradio-facing service.
- `bot.py` — Discord interactions endpoint and bot frontend.
- `tasks/run_backend_modal.sh`, `tasks/run_frontend_modal.sh` — Modal deployment execution paths.

#### Request-to-Answer Pipeline
The online path accepts a user question, retrieves semantically relevant corpus chunks, builds an LLM prompt, and returns a generated response. The central application boundary is between request adapters (`app.py`, `bot.py`) and retrieval/generation modules (`vecstore.py`, `prompts.py`).

- `app.py`
- `bot.py`
- `prompts.py`

#### Document Store: Persistent Corpus
MongoDB Atlas stores the cleaned, structured source corpus and is the durable document-system boundary. Corpus metadata/source inventories are represented in local JSON inputs, while document access is encapsulated by the document-store layer.

- `docstore.py`
- `data/lectures-2022.json`
- `data/llm-papers.json`

#### Vector Retrieval Index
FAISS provides low-latency similarity search over document embeddings. `vecstore.py` is the retrieval abstraction that bridges application queries to the vector index; it should remain decoupled from Discord/UI concerns and from ETL extraction details.

- `vecstore.py`
- `docstore.py`
- `utils.py`

#### Prompting and LLM Invocation
LangChain organizes LLM calls and prompt composition. Prompt definitions are separated into `prompts.py`, allowing retrieval context and user input to be assembled consistently before sending requests to OpenAI. OpenAI is the cost-sensitive external generation dependency.

- `prompts.py`
- `app.py`
- `.env.example`

#### Offline ETL and Index Refresh
Batch ETL converts source materials into cleaned documents suitable for MongoDB and embedding/index construction. Format-specific extractors handle Markdown, PDFs, and videos, with shared transformation logic in the ETL package. Modal also executes these batch jobs.

- `etl/markdown.py`
- `etl/pdfs.py`
- `etl/videos.py`
- `etl/shared.py`
- `Running ETL to Build the Document Corpus.ipynb`

#### Deployment, Integrations, and Observability
Modal hosts both serverless application endpoints and batch processing. Discord invokes the bot through interactions endpoints. Gantry monitors production model behavior and gathers user feedback. MongoDB Atlas, OpenAI, Discord, Modal, and Gantry are the principal external-service boundaries.

- `tasks/run_etl.sh`
- `tasks/setup_environment_modal.sh`
- `tasks/send_secrets_to_modal.sh`
- `setup/mongodb`
- `setup/openai`

#### Operational Flow
1. ETL ingests educational sources, cleans/chunks them, and persists documents to MongoDB.
2. Embeddings/index refresh produces a FAISS retrieval artifact.
3. Discord or Gradio submits a question to the Modal-hosted backend.
4. The backend retrieves relevant chunks, applies prompt templates, and calls OpenAI.
5. The generated answer is returned to the requesting frontend; Gantry records production behavior/feedback.

Main entry points: [`bot.py`](https://github.com/ysn-b13/ask-fsdl-2026/blob/main/bot.py) (Discord bot), [`vecstore.py`](https://github.com/ysn-b13/ask-fsdl-2026/blob/main/vecstore.py) (Vector retrieval), [`app.py`](https://github.com/ysn-b13/ask-fsdl-2026/blob/main/app.py) (Gradio app backend), [`etl/markdown.py`](https://github.com/ysn-b13/ask-fsdl-2026/blob/main/etl/markdown.py) (Format extractors)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_clients["Question channels"]
  node_bot["Discord bot<br/>request adapter<br/>[bot.py]"]
  node_gradio_app["Gradio app backend<br/>request adapter<br/>[app.py]"]
end

subgraph group_online["Online QA"]
  node_vecstore["Vector retrieval<br/>retrieval abstraction<br/>[vecstore.py]"]
  node_prompts["Prompt templates<br/>prompt composition<br/>[prompts.py]"]
end

subgraph group_corpus["Corpus and retrieval"]
  node_faiss[("FAISS index<br/>embedding index")]
  node_docstore["Document store layer<br/>corpus access<br/>[docstore.py]"]
  node_lecture_inventory["Lecture inventory<br/>source metadata<br/>[lectures-2022.json]"]
  node_papers_inventory["Paper inventory<br/>source metadata<br/>[llm-papers.json]"]
end

subgraph group_etl["Offline ingestion"]
  node_etl_extractors["Format extractors<br/>Markdown, PDF, video ETL<br/>[markdown.py]"]
  node_etl_shared["Clean and chunk<br/>shared ETL transform<br/>[shared.py]"]
  node_etl_notebook["Corpus build workflow<br/>batch workflow"]
  node_modal_etl{{"Modal ETL runner<br/>batch execution<br/>[run_etl.sh]"}}
end

subgraph group_external["External services"]
  node_discord{{"Discord<br/>external service"}}
  node_modal_online{{"Modal online runtime<br/>serverless host"}}
  node_openai{{"OpenAI<br/>LLM API"}}
  node_gantry{{"Gantry<br/>model observability"}}
  node_mongodb[("MongoDB Atlas<br/>durable corpus store")]
end

node_discord -->|"interactions"| node_bot
node_bot -->|"runs on"| node_modal_online
node_gradio_app -->|"runs on"| node_modal_online
node_bot -->|"question"| node_vecstore
node_gradio_app -->|"question"| node_vecstore
node_vecstore -->|"similarity search"| node_faiss
node_vecstore -->|"fetch chunks"| node_docstore
node_docstore -->|"read corpus"| node_mongodb
node_vecstore -->|"retrieval context"| node_prompts
node_prompts -->|"prompt and question"| node_openai
node_openai -->|"answer"| node_bot
node_openai -->|"answer"| node_gradio_app
node_bot -.->|"behavior and feedback"| node_gantry
node_gradio_app -.->|"behavior and feedback"| node_gantry
node_lecture_inventory -->|"source list"| node_etl_extractors
node_papers_inventory -->|"source list"| node_etl_extractors
node_etl_extractors -->|"extracted content"| node_etl_shared
node_etl_shared -->|"clean documents"| node_docstore
node_etl_shared -->|"embeddings and refresh"| node_faiss
node_etl_notebook -.->|"orchestrates corpus build"| node_etl_extractors
node_modal_etl -->|"executes batch workflow"| node_etl_notebook

click node_bot "https://github.com/ysn-b13/ask-fsdl-2026/blob/main/bot.py"
click node_gradio_app "https://github.com/ysn-b13/ask-fsdl-2026/blob/main/app.py"
click node_modal_online "https://github.com/ysn-b13/ask-fsdl-2026/blob/main/tasks/run_backend_modal.sh"
click node_vecstore "https://github.com/ysn-b13/ask-fsdl-2026/blob/main/vecstore.py"
click node_prompts "https://github.com/ysn-b13/ask-fsdl-2026/blob/main/prompts.py"
click node_docstore "https://github.com/ysn-b13/ask-fsdl-2026/blob/main/docstore.py"
click node_lecture_inventory "https://github.com/ysn-b13/ask-fsdl-2026/blob/main/data/lectures-2022.json"
click node_papers_inventory "https://github.com/ysn-b13/ask-fsdl-2026/blob/main/data/llm-papers.json"
click node_etl_extractors "https://github.com/ysn-b13/ask-fsdl-2026/blob/main/etl/markdown.py"
click node_etl_shared "https://github.com/ysn-b13/ask-fsdl-2026/blob/main/etl/shared.py"
click node_etl_notebook "https://github.com/ysn-b13/ask-fsdl-2026/blob/main/Running%20ETL%20to%20Build%20the%20Document%20Corpus.ipynb"
click node_modal_etl "https://github.com/ysn-b13/ask-fsdl-2026/blob/main/tasks/run_etl.sh"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_bot,node_gradio_app toneBlue
class node_vecstore,node_prompts toneAmber
class node_faiss,node_docstore,node_lecture_inventory,node_papers_inventory toneMint
class node_etl_extractors,node_etl_shared,node_etl_notebook,node_modal_etl toneRose
class node_discord,node_modal_online,node_openai,node_gantry,node_mongodb toneIndigo
```

## Components

### Question channels

- [Discord bot](https://github.com/ysn-b13/ask-fsdl-2026/blob/main/bot.py): request adapter · `bot.py`
- [Gradio app backend](https://github.com/ysn-b13/ask-fsdl-2026/blob/main/app.py): request adapter · `app.py`

### Online QA

- [Vector retrieval](https://github.com/ysn-b13/ask-fsdl-2026/blob/main/vecstore.py): retrieval abstraction · `vecstore.py`
- [Prompt templates](https://github.com/ysn-b13/ask-fsdl-2026/blob/main/prompts.py): prompt composition · `prompts.py`

### Corpus and retrieval

- **FAISS index**: embedding index · Low-latency similarity-search artifact.
- [Document store layer](https://github.com/ysn-b13/ask-fsdl-2026/blob/main/docstore.py): corpus access · `docstore.py`
- [Lecture inventory](https://github.com/ysn-b13/ask-fsdl-2026/blob/main/data/lectures-2022.json): source metadata · `data/lectures-2022.json`
- [Paper inventory](https://github.com/ysn-b13/ask-fsdl-2026/blob/main/data/llm-papers.json): source metadata · `data/llm-papers.json`

### Offline ingestion

- [Format extractors](https://github.com/ysn-b13/ask-fsdl-2026/blob/main/etl/markdown.py): Markdown, PDF, video ETL · `etl/markdown.py`
- [Clean and chunk](https://github.com/ysn-b13/ask-fsdl-2026/blob/main/etl/shared.py): shared ETL transform · `etl/shared.py`
- [Corpus build workflow](https://github.com/ysn-b13/ask-fsdl-2026/blob/main/Running%20ETL%20to%20Build%20the%20Document%20Corpus.ipynb): batch workflow · `Running ETL to Build the Document Corpus.ipynb`
- [Modal ETL runner](https://github.com/ysn-b13/ask-fsdl-2026/blob/main/tasks/run_etl.sh): batch execution · `tasks/run_etl.sh`

### External services

- **Discord**: external service
- [Modal online runtime](https://github.com/ysn-b13/ask-fsdl-2026/blob/main/tasks/run_backend_modal.sh): serverless host · Hosts application and bot execution paths. · `tasks/run_backend_modal.sh`
- **OpenAI**: LLM API
- **Gantry**: model observability · Records production behavior and user feedback.
- **MongoDB Atlas**: durable corpus store

## Connections

- Discord → Discord bot: interactions
- Discord bot → Modal online runtime: runs on
- Gradio app backend → Modal online runtime: runs on
- Discord bot → Vector retrieval: question
- Gradio app backend → Vector retrieval: question
- Vector retrieval → FAISS index: similarity search
- Vector retrieval → Document store layer: fetch chunks
- Document store layer → MongoDB Atlas: read corpus
- Vector retrieval → Prompt templates: retrieval context
- Prompt templates → OpenAI: prompt and question
- OpenAI → Discord bot: answer
- OpenAI → Gradio app backend: answer
- Discord bot → Gantry: behavior and feedback
- Gradio app backend → Gantry: behavior and feedback
- Lecture inventory → Format extractors: source list
- Paper inventory → Format extractors: source list
- Format extractors → Clean and chunk: extracted content
- Clean and chunk → Document store layer: clean documents
- Clean and chunk → FAISS index: embeddings and refresh
- Corpus build workflow → Format extractors: orchestrates corpus build
- Modal ETL runner → Corpus build workflow: executes batch workflow