How navaneeth-mukundan/chatbot-model works
This repository provides two independent workflows: a FastAPI service that classifies prompts by keyword or returns sentence embeddings, and a recursive scraper that extracts page text, embeds it, and stores it in MongoDB. The scraper’s crawl function owns fetching, extraction, recursion, embedding, and writes; the graph separates those responsibilities to make its data flow legible. The README is empty, and utils.py is empty; the classifier pickle is not referenced by the sampled runtime code.
Main entry points: scraper.py Recursive crawler · main.py FastAPI routes
Components 10
Prompt API
- FastAPI routes
main.py - Keyword classifier
main.py - Prompt embeddings
main.py
Web Ingestion
- Recursive crawler
scraper.py - Page extraction
scraper.py - Text embeddings
scraper.py
Data and Models
- MongoDB collection
scraper.py
Other components
- API client
- Scrape operator
- Web pages
Connections 8
- API client → FastAPI routes: sends requests
main.py - FastAPI routes → Keyword classifier: classifies
main.py - FastAPI routes → Prompt embeddings: encodes prompt
main.py - Scrape operator → Recursive crawler: starts crawl
scraper.py - Recursive crawler → Web pages: fetches pages
scraper.py - Recursive crawler → Page extraction: parses pages
scraper.py - Recursive crawler → Text embeddings: encodes text
scraper.py - Recursive crawler → MongoDB collection: inserts records
scraper.py