# mukul-work/golang-search-engine architecture

> Architecture diagram of the GitHub repository mukul-work/golang-search-engine, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-08). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/mukul-work/golang-search-engine
- Repository: https://github.com/mukul-work/golang-search-engine

## Overview

This Go search engine crawls seed URLs, extracts page content and links, tokenizes text, and stores pages with term frequencies in PostgreSQL; a separate CLI queries that index and prints ranked results. The graph separates crawler coordination, page processing, persistence, and search. Edges reflect the sampled source and README; several crawler helpers and database setup details are represented from their exact tree paths, but their internals were not sampled.

Main entry points: [`crawler/worker.go`](https://github.com/mukul-work/golang-search-engine/blob/main/crawler/worker.go) (Crawler Workers), [`cmd/main.go`](https://github.com/mukul-work/golang-search-engine/blob/main/cmd/main.go) (Crawl CLI), [`cmd/search/main.go`](https://github.com/mukul-work/golang-search-engine/blob/main/cmd/search/main.go) (Search CLI), [`search/query.go`](https://github.com/mukul-work/golang-search-engine/blob/main/search/query.go) (Query Ranking)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_crawl["Crawling"]
  node_crawl_entry["Crawl CLI<br/>[main.go]"]
  node_frontier["URL Frontier<br/>[frontier.go]"]
  node_worker["Crawler Workers<br/>[worker.go]"]
  node_visited[("Visited Set<br/>[visited.go]")]
  node_robots["Robots Cache<br/>[robots.go]"]
  node_limiter["Host Limiter<br/>[limiter.go]"]
end

subgraph group_process["Page Processing"]
  node_fetch["Page Fetch<br/>[get_page_data.go]"]
  node_html_parse["HTML Extraction<br/>[get_html_data.go]"]
  node_tokenizer["Tokenizer<br/>[tokenizer.go]"]
end

subgraph group_storage["Index Storage"]
  node_db_connect["DB Connection<br/>[connect.go]"]
  node_page_index["Page Indexing<br/>[indexPage.go]"]
  node_postgres[("PostgreSQL")]
end

subgraph group_search["Search"]
  node_search_cli["Search CLI<br/>[main.go]"]
  node_query["Query Ranking<br/>[query.go]"]
  node_result_model["Search Results<br/>[models.go]"]
end

node_seed_user(("Seed URLs"))
node_search_user(("Search User"))

node_seed_user -->|"provides seeds"| node_crawl_entry
node_crawl_entry -->|"connects"| node_db_connect
node_crawl_entry -->|"seeds frontier"| node_frontier
node_crawl_entry -->|"checks seeds"| node_visited
node_crawl_entry -->|"starts workers"| node_worker
node_worker -->|"consumes and pushes"| node_frontier
node_worker -->|"deduplicates links"| node_visited
node_worker -->|"checks permission"| node_robots
node_worker -->|"waits per host"| node_limiter
node_worker -->|"fetches page"| node_fetch
node_worker -->|"extracts links and text"| node_html_parse
node_worker -->|"tokenizes text"| node_tokenizer
node_worker -->|"indexes page"| node_page_index
node_page_index -->|"writes page and terms"| node_postgres
node_search_user -->|"enters query"| node_search_cli
node_search_cli -->|"connects"| node_db_connect
node_search_cli -->|"runs query"| node_query
node_query -->|"tokenizes query"| node_tokenizer
node_query -->|"reads and ranks matches"| node_postgres
node_query -->|"builds results"| node_result_model
node_search_cli -->|"prints results"| node_result_model

click node_crawl_entry "https://github.com/mukul-work/golang-search-engine/blob/main/cmd/main.go"
click node_frontier "https://github.com/mukul-work/golang-search-engine/blob/main/crawler/frontier.go"
click node_worker "https://github.com/mukul-work/golang-search-engine/blob/main/crawler/worker.go"
click node_visited "https://github.com/mukul-work/golang-search-engine/blob/main/crawler/visited.go"
click node_robots "https://github.com/mukul-work/golang-search-engine/blob/main/crawler/robots.go"
click node_limiter "https://github.com/mukul-work/golang-search-engine/blob/main/crawler/limiter.go"
click node_fetch "https://github.com/mukul-work/golang-search-engine/blob/main/crawler/get_page_data.go"
click node_html_parse "https://github.com/mukul-work/golang-search-engine/blob/main/crawler/get_html_data.go"
click node_tokenizer "https://github.com/mukul-work/golang-search-engine/blob/main/indexer/tokenizer.go"
click node_db_connect "https://github.com/mukul-work/golang-search-engine/blob/main/db/connect.go"
click node_page_index "https://github.com/mukul-work/golang-search-engine/blob/main/db/indexPage.go"
click node_search_cli "https://github.com/mukul-work/golang-search-engine/blob/main/cmd/search/main.go"
click node_query "https://github.com/mukul-work/golang-search-engine/blob/main/search/query.go"
click node_result_model "https://github.com/mukul-work/golang-search-engine/blob/main/models/models.go"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_crawl_entry,node_frontier,node_worker,node_visited,node_robots,node_limiter,node_search_user toneBlue
class node_fetch,node_html_parse,node_tokenizer toneAmber
class node_db_connect,node_page_index,node_postgres toneMint
class node_search_cli,node_query,node_result_model toneRose
class node_seed_user toneIndigo
```

## Components

### Crawling

- [Crawl CLI](https://github.com/mukul-work/golang-search-engine/blob/main/cmd/main.go): `cmd/main.go`
- [URL Frontier](https://github.com/mukul-work/golang-search-engine/blob/main/crawler/frontier.go): `crawler/frontier.go`
- [Crawler Workers](https://github.com/mukul-work/golang-search-engine/blob/main/crawler/worker.go): `crawler/worker.go`
- [Visited Set](https://github.com/mukul-work/golang-search-engine/blob/main/crawler/visited.go): `crawler/visited.go`
- [Robots Cache](https://github.com/mukul-work/golang-search-engine/blob/main/crawler/robots.go): `crawler/robots.go`
- [Host Limiter](https://github.com/mukul-work/golang-search-engine/blob/main/crawler/limiter.go): `crawler/limiter.go`

### Page Processing

- [Page Fetch](https://github.com/mukul-work/golang-search-engine/blob/main/crawler/get_page_data.go): `crawler/get_page_data.go`
- [HTML Extraction](https://github.com/mukul-work/golang-search-engine/blob/main/crawler/get_html_data.go): `crawler/get_html_data.go`
- [Tokenizer](https://github.com/mukul-work/golang-search-engine/blob/main/indexer/tokenizer.go): `indexer/tokenizer.go`

### Index Storage

- [DB Connection](https://github.com/mukul-work/golang-search-engine/blob/main/db/connect.go): `db/connect.go`
- [Page Indexing](https://github.com/mukul-work/golang-search-engine/blob/main/db/indexPage.go): `db/indexPage.go`
- **PostgreSQL**

### Search

- [Search CLI](https://github.com/mukul-work/golang-search-engine/blob/main/cmd/search/main.go): `cmd/search/main.go`
- [Query Ranking](https://github.com/mukul-work/golang-search-engine/blob/main/search/query.go): `search/query.go`
- [Search Results](https://github.com/mukul-work/golang-search-engine/blob/main/models/models.go): `models/models.go`

### Other components

- **Seed URLs**
- **Search User**

## Connections

- Seed URLs → Crawl CLI: provides seeds (evidence: `cmd/main.go`)
- Crawl CLI → DB Connection: connects (evidence: `cmd/main.go`)
- Crawl CLI → URL Frontier: seeds frontier (evidence: `cmd/main.go`)
- Crawl CLI → Visited Set: checks seeds (evidence: `cmd/main.go`)
- Crawl CLI → Crawler Workers: starts workers (evidence: `cmd/main.go`)
- Crawler Workers → URL Frontier: consumes and pushes (evidence: `crawler/worker.go`)
- Crawler Workers → Visited Set: deduplicates links (evidence: `crawler/worker.go`)
- Crawler Workers → Robots Cache: checks permission (evidence: `crawler/worker.go`)
- Crawler Workers → Host Limiter: waits per host (evidence: `crawler/worker.go`)
- Crawler Workers → Page Fetch: fetches page (evidence: `crawler/worker.go`)
- Crawler Workers → HTML Extraction: extracts links and text (evidence: `crawler/worker.go`)
- Crawler Workers → Tokenizer: tokenizes text (evidence: `crawler/worker.go`)
- Crawler Workers → Page Indexing: indexes page (evidence: `crawler/worker.go`)
- Page Indexing → PostgreSQL: writes page and terms (evidence: `db/indexPage.go`)
- Search User → Search CLI: enters query (evidence: `cmd/search/main.go`)
- Search CLI → DB Connection: connects (evidence: `cmd/search/main.go`)
- Search CLI → Query Ranking: runs query (evidence: `cmd/search/main.go`)
- Query Ranking → Tokenizer: tokenizes query (evidence: `search/query.go`)
- Query Ranking → PostgreSQL: reads and ranks matches (evidence: `search/query.go`)
- Query Ranking → Search Results: builds results (evidence: `search/query.go`)
- Search CLI → Search Results: prints results (evidence: `cmd/search/main.go`)
