How mukul-work/golang-search-engine works
This Go search engine crawls seed URLs, extracts page content and links, tokenizes text, and stores pages with term frequencies in PostgreSQL; a separate CLI queries that index and prints ranked results. The graph separates crawler coordination, page processing, persistence, and search. Edges reflect the sampled source and README; several crawler helpers and database setup details are represented from their exact tree paths, but their internals were not sampled.
Main entry points: worker.go Crawler Workers · cmd/main.go Crawl CLI · search/main.go Search CLI · query.go Query Ranking
Components 17
Crawling
- Crawl CLI
cmd/main.go - URL Frontier
crawler/frontier.go - Crawler Workers
crawler/worker.go - Visited Set
crawler/visited.go - Robots Cache
crawler/robots.go - Host Limiter
crawler/limiter.go
Page Processing
- Page Fetch
crawler/get_page_data.go - HTML Extraction
crawler/get_html_data.go - Tokenizer
indexer/tokenizer.go
Index Storage
- DB Connection
db/connect.go - Page Indexing
db/indexPage.go - PostgreSQL
Search
- Search CLI
cmd/search/main.go - Query Ranking
search/query.go - Search Results
models/models.go
Other components
- Seed URLs
- Search User
Connections 21
- Seed URLs → Crawl CLI: provides seeds
cmd/main.go - Crawl CLI → DB Connection: connects
cmd/main.go - Crawl CLI → URL Frontier: seeds frontier
cmd/main.go - Crawl CLI → Visited Set: checks seeds
cmd/main.go - Crawl CLI → Crawler Workers: starts workers
cmd/main.go - Crawler Workers → URL Frontier: consumes and pushes
crawler/worker.go - Crawler Workers → Visited Set: deduplicates links
crawler/worker.go - Crawler Workers → Robots Cache: checks permission
crawler/worker.go - Crawler Workers → Host Limiter: waits per host
crawler/worker.go - Crawler Workers → Page Fetch: fetches page
crawler/worker.go - Crawler Workers → HTML Extraction: extracts links and text
crawler/worker.go - Crawler Workers → Tokenizer: tokenizes text
crawler/worker.go - Crawler Workers → Page Indexing: indexes page
crawler/worker.go - Page Indexing → PostgreSQL: writes page and terms
db/indexPage.go - Search User → Search CLI: enters query
cmd/search/main.go - Search CLI → DB Connection: connects
cmd/search/main.go - Search CLI → Query Ranking: runs query
cmd/search/main.go - Query Ranking → Tokenizer: tokenizes query
search/query.go - Query Ranking → PostgreSQL: reads and ranks matches
search/query.go - Query Ranking → Search Results: builds results
search/query.go - Search CLI → Search Results: prints results
cmd/search/main.go