# reyna100/searchengine architecture

> Architecture diagram of the GitHub repository reyna100/searchengine, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-07). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/reyna100/searchengine
- Repository: https://github.com/reyna100/searchengine

## Overview

This is an educational full-stack web search engine. Users submit queries in the React UI, which calls the Express API; the search engine retrieves and ranks indexed documents and returns snippets and paginated results. A separate crawl workflow accepts a seed URL, checks robots rules, fetches and parses HTML, discovers links, and indexes pages. The README also documents API health/statistics and optional JSON persistence. Relationships among sampled modules are shown from source; unsampled subsystem links are retained only where documented by the README.

Main entry points: [`backend/src/search/searchEngine.js`](https://github.com/reyna100/searchengine/blob/main/backend/src/search/searchEngine.js) (Search engine), [`backend/src/crawler/crawler.js`](https://github.com/reyna100/searchengine/blob/main/backend/src/crawler/crawler.js) (Crawl orchestrator), [`frontend/src/App.jsx`](https://github.com/reyna100/searchengine/blob/main/frontend/src/App.jsx) (Search application), [`backend/src/server.js`](https://github.com/reyna100/searchengine/blob/main/backend/src/server.js) (Express server)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_ui["Search interface"]
  node_app["Search application<br/>[App.jsx]"]
  node_searchbar["Search input<br/>[SearchBar.jsx]"]
  node_results["Result list<br/>[ResultsList.jsx]"]
  node_pagination["Pagination<br/>[Pagination.jsx]"]
  node_crawlerpanel["Crawler panel<br/>[CrawlerPanel.jsx]"]
  node_searchclient["Search API client<br/>[searchApi.js]"]
  node_crawlclient["Crawler API client<br/>[crawlerApi.js]"]
end

subgraph group_api["HTTP API"]
  node_server["Express server<br/>[server.js]"]
  node_searchroutes["Search routes<br/>[searchRoutes.js]"]
  node_searchcontroller["Search handler"]
  node_crawlroutes["Crawl routes<br/>[crawlerRoutes.js]"]
  node_crawlcontroller["Crawl handler"]
  node_statsroutes["Stats routes<br/>[statsRoutes.js]"]
  node_statscontroller["Stats and health<br/>[statsController.js]"]
end

subgraph group_crawl["Web crawling"]
  node_crawler["Crawl orchestrator<br/>[crawler.js]"]
  node_urlqueue["URL queue<br/>[urlQueue.js]"]
  node_normalizer["URL normalizer<br/>[urlNormalizer.js]"]
  node_robots["Robots checker<br/>[robotsChecker.js]"]
  node_parser["HTML parser<br/>[htmlParser.js]"]
end

subgraph group_search["Index and retrieval"]
  node_searchengine["Search engine<br/>[searchEngine.js]"]
  node_tokenizer["Tokenizer<br/>[tokenizer.js]"]
  node_inverted["Inverted index<br/>[invertedIndex.js]"]
  node_bm25["BM25 ranking<br/>[bm25.js]"]
  node_snippets["Snippet generator"]
  node_store[("Document store<br/>[indexStore.js]")]
  node_seed["Seed documents<br/>[seedData.js]"]
end

node_user(("Search user"))
node_web(("Public websites"))

node_user -->|"uses"| node_app
node_app -->|"renders"| node_searchbar
node_app -->|"renders"| node_results
node_app -->|"renders"| node_pagination
node_app -->|"renders"| node_crawlerpanel
node_app -->|"requests search"| node_searchclient
node_searchclient -->|"fetches API"| node_server
node_crawlerpanel -->|"starts crawl"| node_crawlclient
node_crawlerpanel -->|"fetches stats"| node_searchclient
node_server -->|"mounts"| node_searchroutes
node_server -->|"mounts"| node_crawlroutes
node_server -->|"mounts"| node_statsroutes
node_server -->|"seeds documents"| node_searchengine
node_searchroutes -->|"routes to"| node_searchcontroller
node_searchcontroller -->|"searches"| node_searchengine
node_crawlroutes -->|"routes to"| node_crawlcontroller
node_crawlcontroller -->|"starts crawl"| node_crawler
node_statsroutes -->|"routes to"| node_statscontroller
node_statscontroller -->|"reads stats"| node_searchengine
node_statscontroller -->|"reads status"| node_crawler
node_crawler -->|"dispatches URLs"| node_urlqueue
node_crawler -->|"normalizes URLs"| node_normalizer
node_crawler -->|"checks policy"| node_robots
node_crawler -.->|"fetches pages"| node_web
node_crawler -->|"parses HTML"| node_parser
node_parser -->|"normalizes links"| node_normalizer
node_crawler -->|"indexes documents"| node_searchengine
node_searchengine -->|"tokenizes text"| node_tokenizer
node_searchengine -->|"updates and retrieves"| node_inverted
node_searchengine -->|"scores candidates"| node_bm25
node_searchengine -->|"generates snippets"| node_snippets
node_searchengine -->|"stores documents"| node_store

click node_app "https://github.com/reyna100/searchengine/blob/main/frontend/src/App.jsx"
click node_searchbar "https://github.com/reyna100/searchengine/blob/main/frontend/src/components/SearchBar.jsx"
click node_results "https://github.com/reyna100/searchengine/blob/main/frontend/src/components/ResultsList.jsx"
click node_pagination "https://github.com/reyna100/searchengine/blob/main/frontend/src/components/Pagination.jsx"
click node_crawlerpanel "https://github.com/reyna100/searchengine/blob/main/frontend/src/components/CrawlerPanel.jsx"
click node_searchclient "https://github.com/reyna100/searchengine/blob/main/frontend/src/api/searchApi.js"
click node_crawlclient "https://github.com/reyna100/searchengine/blob/main/frontend/src/api/crawlerApi.js"
click node_server "https://github.com/reyna100/searchengine/blob/main/backend/src/server.js"
click node_searchroutes "https://github.com/reyna100/searchengine/blob/main/backend/src/routes/searchRoutes.js"
click node_searchcontroller "https://github.com/reyna100/searchengine/blob/main/backend/src/controllers/searchController.js"
click node_crawlroutes "https://github.com/reyna100/searchengine/blob/main/backend/src/routes/crawlerRoutes.js"
click node_crawlcontroller "https://github.com/reyna100/searchengine/blob/main/backend/src/controllers/crawlerController.js"
click node_statsroutes "https://github.com/reyna100/searchengine/blob/main/backend/src/routes/statsRoutes.js"
click node_statscontroller "https://github.com/reyna100/searchengine/blob/main/backend/src/controllers/statsController.js"
click node_crawler "https://github.com/reyna100/searchengine/blob/main/backend/src/crawler/crawler.js"
click node_urlqueue "https://github.com/reyna100/searchengine/blob/main/backend/src/crawler/urlQueue.js"
click node_normalizer "https://github.com/reyna100/searchengine/blob/main/backend/src/crawler/urlNormalizer.js"
click node_robots "https://github.com/reyna100/searchengine/blob/main/backend/src/crawler/robotsChecker.js"
click node_parser "https://github.com/reyna100/searchengine/blob/main/backend/src/crawler/htmlParser.js"
click node_searchengine "https://github.com/reyna100/searchengine/blob/main/backend/src/search/searchEngine.js"
click node_tokenizer "https://github.com/reyna100/searchengine/blob/main/backend/src/search/tokenizer.js"
click node_inverted "https://github.com/reyna100/searchengine/blob/main/backend/src/search/invertedIndex.js"
click node_bm25 "https://github.com/reyna100/searchengine/blob/main/backend/src/search/bm25.js"
click node_snippets "https://github.com/reyna100/searchengine/blob/main/backend/src/search/snippetGenerator.js"
click node_store "https://github.com/reyna100/searchengine/blob/main/backend/src/storage/indexStore.js"
click node_seed "https://github.com/reyna100/searchengine/blob/main/backend/src/utils/seedData.js"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_app,node_searchbar,node_results,node_pagination,node_crawlerpanel,node_searchclient,node_crawlclient,node_user toneBlue
class node_server,node_searchroutes,node_searchcontroller,node_crawlroutes,node_crawlcontroller,node_statsroutes,node_statscontroller toneAmber
class node_crawler,node_urlqueue,node_normalizer,node_robots,node_parser toneMint
class node_searchengine,node_tokenizer,node_inverted,node_bm25,node_snippets,node_store,node_seed toneRose
class node_web toneIndigo
```

## Components

### Search interface

- [Search application](https://github.com/reyna100/searchengine/blob/main/frontend/src/App.jsx): `frontend/src/App.jsx`
- [Search input](https://github.com/reyna100/searchengine/blob/main/frontend/src/components/SearchBar.jsx): `frontend/src/components/SearchBar.jsx`
- [Result list](https://github.com/reyna100/searchengine/blob/main/frontend/src/components/ResultsList.jsx): `frontend/src/components/ResultsList.jsx`
- [Pagination](https://github.com/reyna100/searchengine/blob/main/frontend/src/components/Pagination.jsx): `frontend/src/components/Pagination.jsx`
- [Crawler panel](https://github.com/reyna100/searchengine/blob/main/frontend/src/components/CrawlerPanel.jsx): `frontend/src/components/CrawlerPanel.jsx`
- [Search API client](https://github.com/reyna100/searchengine/blob/main/frontend/src/api/searchApi.js): `frontend/src/api/searchApi.js`
- [Crawler API client](https://github.com/reyna100/searchengine/blob/main/frontend/src/api/crawlerApi.js): `frontend/src/api/crawlerApi.js`

### HTTP API

- [Express server](https://github.com/reyna100/searchengine/blob/main/backend/src/server.js): `backend/src/server.js`
- [Search routes](https://github.com/reyna100/searchengine/blob/main/backend/src/routes/searchRoutes.js): `backend/src/routes/searchRoutes.js`
- [Search handler](https://github.com/reyna100/searchengine/blob/main/backend/src/controllers/searchController.js): `backend/src/controllers/searchController.js`
- [Crawl routes](https://github.com/reyna100/searchengine/blob/main/backend/src/routes/crawlerRoutes.js): `backend/src/routes/crawlerRoutes.js`
- [Crawl handler](https://github.com/reyna100/searchengine/blob/main/backend/src/controllers/crawlerController.js): `backend/src/controllers/crawlerController.js`
- [Stats routes](https://github.com/reyna100/searchengine/blob/main/backend/src/routes/statsRoutes.js): `backend/src/routes/statsRoutes.js`
- [Stats and health](https://github.com/reyna100/searchengine/blob/main/backend/src/controllers/statsController.js): `backend/src/controllers/statsController.js`

### Web crawling

- [Crawl orchestrator](https://github.com/reyna100/searchengine/blob/main/backend/src/crawler/crawler.js): `backend/src/crawler/crawler.js`
- [URL queue](https://github.com/reyna100/searchengine/blob/main/backend/src/crawler/urlQueue.js): `backend/src/crawler/urlQueue.js`
- [URL normalizer](https://github.com/reyna100/searchengine/blob/main/backend/src/crawler/urlNormalizer.js): `backend/src/crawler/urlNormalizer.js`
- [Robots checker](https://github.com/reyna100/searchengine/blob/main/backend/src/crawler/robotsChecker.js): `backend/src/crawler/robotsChecker.js`
- [HTML parser](https://github.com/reyna100/searchengine/blob/main/backend/src/crawler/htmlParser.js): `backend/src/crawler/htmlParser.js`

### Index and retrieval

- [Search engine](https://github.com/reyna100/searchengine/blob/main/backend/src/search/searchEngine.js): `backend/src/search/searchEngine.js`
- [Tokenizer](https://github.com/reyna100/searchengine/blob/main/backend/src/search/tokenizer.js): `backend/src/search/tokenizer.js`
- [Inverted index](https://github.com/reyna100/searchengine/blob/main/backend/src/search/invertedIndex.js): `backend/src/search/invertedIndex.js`
- [BM25 ranking](https://github.com/reyna100/searchengine/blob/main/backend/src/search/bm25.js): `backend/src/search/bm25.js`
- [Snippet generator](https://github.com/reyna100/searchengine/blob/main/backend/src/search/snippetGenerator.js): `backend/src/search/snippetGenerator.js`
- [Document store](https://github.com/reyna100/searchengine/blob/main/backend/src/storage/indexStore.js): `backend/src/storage/indexStore.js`
- [Seed documents](https://github.com/reyna100/searchengine/blob/main/backend/src/utils/seedData.js): `backend/src/utils/seedData.js`

### Other components

- **Search user**
- **Public websites**

## Connections

- Search user → Search application: uses (evidence: `README.md`)
- Search application → Search input: renders (evidence: `frontend/src/App.jsx`)
- Search application → Result list: renders (evidence: `frontend/src/App.jsx`)
- Search application → Pagination: renders (evidence: `frontend/src/App.jsx`)
- Search application → Crawler panel: renders (evidence: `frontend/src/App.jsx`)
- Search application → Search API client: requests search (evidence: `frontend/src/App.jsx`)
- Search API client → Express server: fetches API (evidence: `frontend/src/api/searchApi.js`)
- Crawler panel → Crawler API client: starts crawl (evidence: `frontend/src/components/CrawlerPanel.jsx`)
- Crawler panel → Search API client: fetches stats (evidence: `frontend/src/components/CrawlerPanel.jsx`)
- Express server → Search routes: mounts (evidence: `backend/src/server.js`)
- Express server → Crawl routes: mounts (evidence: `backend/src/server.js`)
- Express server → Stats routes: mounts (evidence: `backend/src/server.js`)
- Express server → Search engine: seeds documents (evidence: `backend/src/server.js`)
- Search routes → Search handler: routes to (evidence: `backend/src/routes/searchRoutes.js`)
- Search handler → Search engine: searches (evidence: `backend/src/controllers/searchController.js`)
- Crawl routes → Crawl handler: routes to (evidence: `backend/src/routes/crawlerRoutes.js`)
- Crawl handler → Crawl orchestrator: starts crawl (evidence: `backend/src/controllers/crawlerController.js`)
- Stats routes → Stats and health: routes to (evidence: `backend/src/routes/statsRoutes.js`)
- Stats and health → Search engine: reads stats (evidence: `backend/src/controllers/statsController.js`)
- Stats and health → Crawl orchestrator: reads status (evidence: `backend/src/controllers/statsController.js`)
- Crawl orchestrator → URL queue: dispatches URLs (evidence: `backend/src/crawler/crawler.js`)
- Crawl orchestrator → URL normalizer: normalizes URLs (evidence: `backend/src/crawler/crawler.js`)
- Crawl orchestrator → Robots checker: checks policy (evidence: `backend/src/crawler/crawler.js`)
- Crawl orchestrator → Public websites: fetches pages (evidence: `README.md`)
- Crawl orchestrator → HTML parser: parses HTML (evidence: `backend/src/crawler/crawler.js`)
- HTML parser → URL normalizer: normalizes links (evidence: `backend/src/crawler/htmlParser.js`)
- Crawl orchestrator → Search engine: indexes documents (evidence: `backend/src/crawler/crawler.js`)
- Search engine → Tokenizer: tokenizes text (evidence: `backend/src/search/searchEngine.js`)
- Search engine → Inverted index: updates and retrieves (evidence: `backend/src/search/searchEngine.js`)
- Search engine → BM25 ranking: scores candidates (evidence: `backend/src/search/searchEngine.js`)
- Search engine → Snippet generator: generates snippets (evidence: `backend/src/search/searchEngine.js`)
- Search engine → Document store: stores documents (evidence: `backend/src/search/searchEngine.js`)
