How reyna100/searchengine works
This is an educational full-stack web search engine. Users submit queries in the React UI, which calls the Express API; the search engine retrieves and ranks indexed documents and returns snippets and paginated results. A separate crawl workflow accepts a seed URL, checks robots rules, fetches and parses HTML, discovers links, and indexes pages. The README also documents API health/statistics and optional JSON persistence. Relationships among sampled modules are shown from source; unsampled subsystem links are retained only where documented by the README.
Main entry points: searchEngine.js Search engine · crawler.js Crawl orchestrator · App.jsx Search application · server.js Express server
Components 28
Search interface
- Search application
frontend/src/App.jsx - Search input
frontend/src/components/SearchBar.jsx - Result list
frontend/src/components/ResultsList.jsx - Pagination
frontend/src/components/Pagination.jsx - Crawler panel
frontend/src/components/CrawlerPanel.jsx - Search API client
frontend/src/api/searchApi.js - Crawler API client
frontend/src/api/crawlerApi.js
HTTP API
- Express server
backend/src/server.js - Search routes
backend/src/routes/searchRoutes.js - Search handler
backend/src/controllers/searchController.js - Crawl routes
backend/src/routes/crawlerRoutes.js - Crawl handler
backend/src/controllers/crawlerController.js - Stats routes
backend/src/routes/statsRoutes.js - Stats and health
backend/src/controllers/statsController.js
Web crawling
- Crawl orchestrator
backend/src/crawler/crawler.js - URL queue
backend/src/crawler/urlQueue.js - URL normalizer
backend/src/crawler/urlNormalizer.js - Robots checker
backend/src/crawler/robotsChecker.js - HTML parser
backend/src/crawler/htmlParser.js
Index and retrieval
- Search engine
backend/src/search/searchEngine.js - Tokenizer
backend/src/search/tokenizer.js - Inverted index
backend/src/search/invertedIndex.js - BM25 ranking
backend/src/search/bm25.js - Snippet generator
backend/src/search/snippetGenerator.js - Document store
backend/src/storage/indexStore.js - Seed documents
backend/src/utils/seedData.js
Other components
- Search user
- Public websites
Connections 32
- Search user → Search application: uses
README.md - Search application → Search input: renders
frontend/src/App.jsx - Search application → Result list: renders
frontend/src/App.jsx - Search application → Pagination: renders
frontend/src/App.jsx - Search application → Crawler panel: renders
frontend/src/App.jsx - Search application → Search API client: requests search
frontend/src/App.jsx - Search API client → Express server: fetches API
frontend/src/api/searchApi.js - Crawler panel → Crawler API client: starts crawl
frontend/src/components/CrawlerPanel.jsx - Crawler panel → Search API client: fetches stats
frontend/src/components/CrawlerPanel.jsx - Express server → Search routes: mounts
backend/src/server.js - Express server → Crawl routes: mounts
backend/src/server.js - Express server → Stats routes: mounts
backend/src/server.js - Express server → Search engine: seeds documents
backend/src/server.js - Search routes → Search handler: routes to
backend/src/routes/searchRoutes.js - Search handler → Search engine: searches
backend/src/controllers/searchController.js - Crawl routes → Crawl handler: routes to
backend/src/routes/crawlerRoutes.js - Crawl handler → Crawl orchestrator: starts crawl
backend/src/controllers/crawlerController.js - Stats routes → Stats and health: routes to
backend/src/routes/statsRoutes.js - Stats and health → Search engine: reads stats
backend/src/controllers/statsController.js - Stats and health → Crawl orchestrator: reads status
backend/src/controllers/statsController.js - Crawl orchestrator → URL queue: dispatches URLs
backend/src/crawler/crawler.js - Crawl orchestrator → URL normalizer: normalizes URLs
backend/src/crawler/crawler.js - Crawl orchestrator → Robots checker: checks policy
backend/src/crawler/crawler.js - Crawl orchestrator → Public websites: fetches pages
README.md - Crawl orchestrator → HTML parser: parses HTML
backend/src/crawler/crawler.js - HTML parser → URL normalizer: normalizes links
backend/src/crawler/htmlParser.js - Crawl orchestrator → Search engine: indexes documents
backend/src/crawler/crawler.js - Search engine → Tokenizer: tokenizes text
backend/src/search/searchEngine.js - Search engine → Inverted index: updates and retrieves
backend/src/search/searchEngine.js - Search engine → BM25 ranking: scores candidates
backend/src/search/searchEngine.js - Search engine → Snippet generator: generates snippets
backend/src/search/searchEngine.js - Search engine → Document store: stores documents
backend/src/search/searchEngine.js