How afazio1/web-crawler works
This repo is a single-binary Go command-line web crawler with an optional MongoDB-backed “web archive”. It:
• Accepts one (currently fixed) seed URL (per README: future work is prompting user).
• Concurrently fetches pages, parses HTML, extracts a limited content snippet, discovers links, dedupes/avoids loops, and maintains a crawl frontier (BFS per README).
• Persists {url, title, content} into MongoDB, where MongoDB Atlas Search builds an inverted index over title + content for fast querying.
• Reports crawl statistics periodically (every minute): crawl speed and crawled-to-queued ratio.
So the “system” is not a multi-service product; it’s a crawler pipeline (with concurrency) + external dependencies (websites and MongoDB).
What the file tree implies architecturally
Top-level files
• main.go: everything (entrypoint + crawler logic + DB logic + metrics) likely lives here.
• go.mod, go.sum: standard Go module. README says only MongoDB is external; most functionality uses Go stdlib (net/http, html parsing, concurrency primitives, time).
Key architectural implication: This is a monolith process but internally it is best represented as a pipeline with concurrent workers (goroutines) and shared state (queue/frontier, visited set, stats).
User / Execution Environment
• User / Terminal
• Crawler CLI binary (Go program) running locally
Seed Provider
• Currently fixed seed (per README), but represent it as an input source.
Frontier / URL Queue (BFS)
• BFS frontier (queue).
• Track “queued count” for stats.
Deduper / Visited Set
• Ensures “avoids loops and dead ends.”
• Prevents re-enqueue of URLs already seen/crawled.
Fetcher (HTTP client)
• Concurrency: worker pool of goroutines.
• Error handling: invalid URLs, 404s, etc.
HTML Parser
• Extract <title>
• Ignore <script> tags
• Extract first 500 characters after <body> tag for content (as described)
• Extract outgoing links (note: README says it ignores relative links—reflect that decision).
URL Normalizer / Filter
• Drop relative links (explicit in README)
• Possibly enforce scheme/domain filters (if present in code)
Persistence Layer
• MongoDB Insert Worker(s) (README: inserts are concurrent)
• Document schema: { url, title, content }
Metrics / Benchmarking
• Periodic timer (every minute) that computes:
• Pages/sec
• Crawled-to-queued ratio/sec
• Output to CLI (and maybe stored in memory only)
Target Websites (WWW)
• External HTTP servers you crawl.
MongoDB Web Archive
• MongoDB database storing crawled documents.
MongoDB Atlas Search Index
• Logical component attached to MongoDB collection:
• Index on title + content
• Keyword analyzer
• Used by Search Tester GUI (external tool)
(Optional but accurate to README) MongoDB Search Tester GUI
• Not part of code, but part of the system workflow for validating/searching the archive.
Metrics path
• Crawler Core → Metrics Reporter → CLI Output
Search path (post-crawl)
• User → MongoDB Search Tester GUI → MongoDB Atlas Search Index → MongoDB Collection
• This path is important because “web archive” value comes from searchability, not just storage.
Even for a small repo, explicitly label these patterns because they’re the “essence”:
Pipeline / staged processing
• Fetch → Parse → Extract → Persist
• Link extraction loops back into Frontier
Concurrent worker pool
• Fetchers and DB inserters run concurrently
• Use a “worker pool” label even if implementation is goroutines reading from channels.
BFS crawler frontier
• Queue as BFS, not stack/DFS
Open main.go and identify
• Where the URL queue is implemented (slice/chan/queue struct)
• Where visited/dedup state is maintained (map/set)
• The worker counts and channel topology (if any)
• MongoDB client init + insert calls
• Metrics ticker interval and what it logs
• If DB inserts happen on the same goroutines as fetching/parsing, collapse “DB writer pool” into “Fetcher/Parser worker.”
A good rule: prefer representing concurrency boundaries that exist in code (channels/worker pools) instead of hypothetical layers.
“Don’t overthink it” — but do separate into many components
• Seed input
• Frontier
• Dedup
• Fetch
• Parse
• Extract content/title
• Extract links
• Filter/normalize
• Persist
• Search index
• Metrics
That decomposition is what makes the design reviewable and makes future enhancements (different crawl algorithms, better content extraction, relative link handling) obvious.