# navaneeth-mukundan/chatbot-model architecture

> Architecture diagram of the GitHub repository navaneeth-mukundan/chatbot-model, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-21). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/navaneeth-mukundan/chatbot-model
- Repository: https://github.com/navaneeth-mukundan/chatbot-model

## Overview

This repository provides two independent workflows: a FastAPI service that classifies prompts by keyword or returns sentence embeddings, and a recursive scraper that extracts page text, embeds it, and stores it in MongoDB. The scraper’s crawl function owns fetching, extraction, recursion, embedding, and writes; the graph separates those responsibilities to make its data flow legible. The README is empty, and `utils.py` is empty; the classifier pickle is not referenced by the sampled runtime code.

Main entry points: [`scraper.py`](https://github.com/navaneeth-mukundan/chatbot-model/blob/main/scraper.py) (Recursive crawler), [`main.py`](https://github.com/navaneeth-mukundan/chatbot-model/blob/main/main.py) (FastAPI routes)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_api["Prompt API"]
  node_api["FastAPI routes<br/>[main.py]"]
  node_classifier["Keyword classifier<br/>[main.py]"]
  node_api_embeddings["Prompt embeddings<br/>[main.py]"]
end

subgraph group_ingestion["Web Ingestion"]
  node_crawler["Recursive crawler<br/>[scraper.py]"]
  node_page_extraction["Page extraction<br/>[scraper.py]"]
  node_scrape_embeddings["Text embeddings<br/>[scraper.py]"]
end

subgraph group_storage["Data and Models"]
  node_mongo[("MongoDB collection<br/>[scraper.py]")]
end

node_client(("API client"))
node_operator(("Scrape operator"))
node_websites["Web pages"]

node_client -->|"sends requests"| node_api
node_api -->|"classifies"| node_classifier
node_api -->|"encodes prompt"| node_api_embeddings
node_operator -->|"starts crawl"| node_crawler
node_crawler -->|"fetches pages"| node_websites
node_crawler -->|"parses pages"| node_page_extraction
node_crawler -->|"encodes text"| node_scrape_embeddings
node_crawler -->|"inserts records"| node_mongo

click node_api "https://github.com/navaneeth-mukundan/chatbot-model/blob/main/main.py"
click node_classifier "https://github.com/navaneeth-mukundan/chatbot-model/blob/main/main.py"
click node_api_embeddings "https://github.com/navaneeth-mukundan/chatbot-model/blob/main/main.py"
click node_crawler "https://github.com/navaneeth-mukundan/chatbot-model/blob/main/scraper.py"
click node_page_extraction "https://github.com/navaneeth-mukundan/chatbot-model/blob/main/scraper.py"
click node_scrape_embeddings "https://github.com/navaneeth-mukundan/chatbot-model/blob/main/scraper.py"
click node_mongo "https://github.com/navaneeth-mukundan/chatbot-model/blob/main/scraper.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_api,node_classifier,node_api_embeddings,node_client toneBlue
class node_crawler,node_page_extraction,node_scrape_embeddings toneAmber
class node_mongo toneMint
class node_operator,node_websites toneIndigo
```

## Components

### Prompt API

- [FastAPI routes](https://github.com/navaneeth-mukundan/chatbot-model/blob/main/main.py): `main.py`
- [Keyword classifier](https://github.com/navaneeth-mukundan/chatbot-model/blob/main/main.py): `main.py`
- [Prompt embeddings](https://github.com/navaneeth-mukundan/chatbot-model/blob/main/main.py): `main.py`

### Web Ingestion

- [Recursive crawler](https://github.com/navaneeth-mukundan/chatbot-model/blob/main/scraper.py): `scraper.py`
- [Page extraction](https://github.com/navaneeth-mukundan/chatbot-model/blob/main/scraper.py): `scraper.py`
- [Text embeddings](https://github.com/navaneeth-mukundan/chatbot-model/blob/main/scraper.py): `scraper.py`

### Data and Models

- [MongoDB collection](https://github.com/navaneeth-mukundan/chatbot-model/blob/main/scraper.py): `scraper.py`

### Other components

- **API client**
- **Scrape operator**
- **Web pages**

## Connections

- API client → FastAPI routes: sends requests (evidence: `main.py`)
- FastAPI routes → Keyword classifier: classifies (evidence: `main.py`)
- FastAPI routes → Prompt embeddings: encodes prompt (evidence: `main.py`)
- Scrape operator → Recursive crawler: starts crawl (evidence: `scraper.py`)
- Recursive crawler → Web pages: fetches pages (evidence: `scraper.py`)
- Recursive crawler → Page extraction: parses pages (evidence: `scraper.py`)
- Recursive crawler → Text embeddings: encodes text (evidence: `scraper.py`)
- Recursive crawler → MongoDB collection: inserts records (evidence: `scraper.py`)
