# afazio1/web-crawler architecture

> Architecture diagram of the GitHub repository afazio1/web-crawler, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-02-26). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/afazio1/web-crawler
- Repository: https://github.com/afazio1/web-crawler

## Overview

This repo is a **single-binary Go command-line web crawler** with an optional **MongoDB-backed “web archive”**. It:
- Accepts one (currently fixed) seed URL (per README: future work is prompting user).
- **Concurrently fetches pages**, parses HTML, extracts a limited content snippet, discovers links, dedupes/avoids loops, and maintains a crawl frontier (BFS per README).
- **Persists `{url, title, content}`** into **MongoDB**, where **MongoDB Atlas Search** builds an inverted index over `title` + `content` for fast querying.
- Reports **crawl statistics periodically** (every minute): crawl speed and crawled-to-queued ratio.

So the “system” is not a multi-service product; it’s a **crawler pipeline** (with concurrency) + **external dependencies** (websites and MongoDB).

##### What the file tree implies architecturally
**Top-level files**
- `main.go`: everything (entrypoint + crawler logic + DB logic + metrics) likely lives here.
- `go.mod`, `go.sum`: standard Go module. README says only MongoDB is external; most functionality uses Go stdlib (net/http, html parsing, concurrency primitives, time).

**Key architectural implication**: This is a **monolith process** but internally it is best represented as a **pipeline with concurrent workers** (goroutines) and shared state (queue/frontier, visited set, stats).

##### User / Execution Environment
- **User / Terminal**
- **Crawler CLI binary (Go program)** running locally

##### Seed Provider
- Currently fixed seed (per README), but represent it as an input source.

##### Frontier / URL Queue (BFS)
- BFS frontier (queue).
- Track “queued count” for stats.

##### Deduper / Visited Set
- Ensures “avoids loops and dead ends.”
- Prevents re-enqueue of URLs already seen/crawled.

##### Fetcher (HTTP client)
- Concurrency: worker pool of goroutines.
- Error handling: invalid URLs, 404s, etc.

##### HTML Parser
- Extract `<title>`
- Ignore `<script>` tags
- Extract first 500 characters after `<body>` tag for `content` (as described)
- Extract outgoing links (note: README says it ignores relative links—reflect that decision).

##### URL Normalizer / Filter
- Drop relative links (explicit in README)
- Possibly enforce scheme/domain filters (if present in code)

##### Persistence Layer
- **MongoDB Insert Worker(s)** (README: inserts are concurrent)
- Document schema: `{ url, title, content }`

##### Metrics / Benchmarking
- Periodic timer (every minute) that computes:
- Pages/sec
- Crawled-to-queued ratio/sec
- Output to CLI (and maybe stored in memory only)

##### Target Websites (WWW)
- External HTTP servers you crawl.

##### MongoDB Web Archive
- MongoDB database storing crawled documents.

##### MongoDB Atlas Search Index
- Logical component attached to MongoDB collection:
- Index on `title` + `content`
- Keyword analyzer
- Used by Search Tester GUI (external tool)

##### (Optional but accurate to README) MongoDB Search Tester GUI
- Not part of code, but part of the system workflow for validating/searching the archive.

##### Metrics path
- **Crawler Core → Metrics Reporter → CLI Output**

##### Search path (post-crawl)
- **User → MongoDB Search Tester GUI → MongoDB Atlas Search Index → MongoDB Collection**
- This path is important because “web archive” value comes from searchability, not just storage.

Even for a small repo, explicitly label these patterns because they’re the “essence”:

##### Pipeline / staged processing
- Fetch → Parse → Extract → Persist
- Link extraction loops back into Frontier

##### Concurrent worker pool
- Fetchers and DB inserters run concurrently
- Use a “worker pool” label even if implementation is goroutines reading from channels.

##### BFS crawler frontier
- Queue as BFS, not stack/DFS

##### Open `main.go` and identify
- Where the URL queue is implemented (slice/chan/queue struct)
- Where visited/dedup state is maintained (map/set)
- The worker counts and channel topology (if any)
- MongoDB client init + insert calls
- Metrics ticker interval and what it logs
- If DB inserts happen on the same goroutines as fetching/parsing, collapse “DB writer pool” into “Fetcher/Parser worker.”

A good rule: **prefer representing concurrency boundaries that exist in code** (channels/worker pools) instead of hypothetical layers.

##### “Don’t overthink it” — but do separate into many components
- Seed input
- Frontier
- Dedup
- Fetch
- Parse
- Extract content/title
- Extract links
- Filter/normalize
- Persist
- Search index
- Metrics

That decomposition is what makes the design reviewable and makes future enhancements (different crawl algorithms, better content extraction, relative link handling) obvious.

## Diagram (Mermaid)

```mermaid
flowchart TD
  %% =======================
  %% Diagram 1: System Context
  %% =======================
  subgraph "System Context"
    U["User (Terminal)"]:::external

    CRAWLER["Go Web Crawler (CLI)\n(single-binary process)"]:::process

    WWW(["Target Websites (WWW)\nHTTP servers"]):::external
    MONGO[("MongoDB Web Archive\nCollection: {url,title,content}")]:::datastore
    ATLAS["MongoDB Atlas Search Index\n(inverted index on title + content)"]:::index
    GUI["MongoDB Search Tester GUI\n(external validation tool)"]:::external

    U -->|"runs_cli"| CRAWLER
    CRAWLER -->|"HTTP_GET"| WWW
    CRAWLER -->|"insert_documents"| MONGO
    ATLAS ---|"indexes_title+content"| MONGO

    U -->|"search_queries"| GUI
    GUI -->|"search"| ATLAS
    ATLAS -->|"query_results"| GUI
  end

  %% ===================================
  %% Diagram 2: Container / Runtime View
  %% ===================================
  subgraph "Go Process: Crawler CLI Binary (main.go)"
    SEED["Seed Provider\n(fixed seed URL)"]:::process
    FRONTIER["Frontier / URL Queue\n(BFS frontier)"]:::queue
    DEDUPE["Deduper / Visited Set\n(avoid loops/re-enqueue)"]:::process

    subgraph "Concurrent Worker Pools"
      FETCH["Fetcher Workers\n(HTTP client goroutines)"]:::worker
      PARSE["HTML Parser\n- extract <title>\n- ignore <script>\n- content: first 500 chars after <body>"]:::process
      LINKX["Link Extractor\n(outgoing links)"]:::process
      FILTER["URL Normalizer / Filter\n(drop relative links)"]:::process
      PERSIST["Persistence Layer\nMongoDB insert worker(s)\n{url,title,content}"]:::worker
    end

    METRICS["Metrics Reporter\n(ticker every 1 min)\npages/sec, crawled:queued"]:::metrics
    OUT["CLI Output\n(stats/logs)"]:::external

    %% Primary crawl pipeline
    SEED -->|"seed_URL"| FRONTIER
    FRONTIER -->|"dequeue_URL"| FETCH
    FETCH -->|"HTTP_GET"| WWW
    WWW -->|"HTML_response_or_error"| FETCH
    FETCH -->|"HTML"| PARSE

    PARSE -->|"title+snippet"| PERSIST
    PERSIST -->|"insert_{url,title,content}"| MONGO

    PARSE -->|"outgoing_links"| LINKX
    LINKX -->|"discovered_links"| FILTER
    FILTER -->|"candidate_URLs"| DEDUPE
    DEDUPE -->|"enqueue_new_URLs"| FRONTIER

    %% Metrics path (control/observability)
    FRONTIER -.->|"queued_count"| METRICS
    FETCH -.->|"crawled_count"| METRICS
    METRICS -.->|"log_pages/sec_and_ratio"| OUT
    U -.->|"reads_output"| OUT
  end

  %% =======================
  %% Diagram 3: One Iteration (Sequence)
  %% =======================
  subgraph "One Crawl Iteration (Conceptual Sequence)"
    direction TB;
    SQ1["Frontier Queue\n(BFS)"]:::queue
    SQ2["Fetcher Worker\n(goroutine)"]:::worker
    SQ3["Target Website\n(HTTP)"]:::external
    SQ4["HTML Parser"]:::process
    SQ5["Link Extractor"]:::process
    SQ6["URL Filter\n(drop relative)"]:::process
    SQ7["Deduper / Visited"]:::process
    SQ8["MongoDB Insert Worker"]:::worker
    SQ9[("MongoDB Collection")]:::datastore
    SQ10["Metrics Ticker\n(1 min)"]:::metrics
    SQ11["CLI Output"]:::external

    SQ1 -->|"dequeue_URL"| SQ2
    SQ2 -->|"HTTP_GET"| SQ3
    SQ3 -->|"HTML_or_error"| SQ2
    SQ2 -->|"HTML"| SQ4
    SQ4 -->|"title+content_snippet"| SQ8
    SQ8 -->|"insert_{url,title,content}"| SQ9

    SQ4 -->|"links"| SQ5
    SQ5 -->|"outgoing_links"| SQ6
    SQ6 -->|"absolute_URLs_only"| SQ7
    SQ7 -->|"enqueue_new_URLs"| SQ1

    SQ10 -.->|"compute_pages/sec_and_crawled:queued"| SQ11
  end

  %% =======================
  %% Notes / Constraints
  %% =======================
  NOTE1["Constraint:\nIgnores relative links"]:::note
  NOTE2["Content extraction:\nfirst 500 chars after <body>\n(<script> tags ignored)"]:::note
  NOTE3["Pattern:\nPipeline + concurrent workers + BFS frontier\n(loopback via link discovery)"]:::note

  FILTER --- NOTE1
  PARSE --- NOTE2
  FRONTIER --- NOTE3

  %% =======================
  %% Click events (component mapping)
  %% =======================
  click CRAWLER "https://github.com/afazio1/web-crawler/blob/main/main.go"
  click SEED "https://github.com/afazio1/web-crawler/blob/main/main.go"
  click FRONTIER "https://github.com/afazio1/web-crawler/blob/main/main.go"
  click DEDUPE "https://github.com/afazio1/web-crawler/blob/main/main.go"
  click FETCH "https://github.com/afazio1/web-crawler/blob/main/main.go"
  click PARSE "https://github.com/afazio1/web-crawler/blob/main/main.go"
  click LINKX "https://github.com/afazio1/web-crawler/blob/main/main.go"
  click FILTER "https://github.com/afazio1/web-crawler/blob/main/main.go"
  click PERSIST "https://github.com/afazio1/web-crawler/blob/main/main.go"
  click METRICS "https://github.com/afazio1/web-crawler/blob/main/main.go"
  click NOTE3 "https://github.com/afazio1/web-crawler/blob/main/README.md"
  click NOTE1 "https://github.com/afazio1/web-crawler/blob/main/README.md"
  click NOTE2 "https://github.com/afazio1/web-crawler/blob/main/README.md"
  click U "https://github.com/afazio1/web-crawler/blob/main/README.md"
  click GUI "https://github.com/afazio1/web-crawler/blob/main/README.md"
  click ATLAS "https://github.com/afazio1/web-crawler/blob/main/README.md"
  click MONGO "https://github.com/afazio1/web-crawler/blob/main/README.md"
  click WWW "https://github.com/afazio1/web-crawler/blob/main/README.md"
  click OUT "https://github.com/afazio1/web-crawler/blob/main/main.go"
  click SQ1 "https://github.com/afazio1/web-crawler/blob/main/main.go"
  click SQ2 "https://github.com/afazio1/web-crawler/blob/main/main.go"
  click SQ4 "https://github.com/afazio1/web-crawler/blob/main/main.go"
  click SQ5 "https://github.com/afazio1/web-crawler/blob/main/main.go"
  click SQ6 "https://github.com/afazio1/web-crawler/blob/main/main.go"
  click SQ7 "https://github.com/afazio1/web-crawler/blob/main/main.go"
  click SQ8 "https://github.com/afazio1/web-crawler/blob/main/main.go"
  click SQ10 "https://github.com/afazio1/web-crawler/blob/main/main.go"
  click SQ11 "https://github.com/afazio1/web-crawler/blob/main/main.go"

  click NOTE3 "https://github.com/afazio1/web-crawler/tree/main/images/"
  click NOTE2 "https://github.com/afazio1/web-crawler/tree/main/images/"
  click NOTE1 "https://github.com/afazio1/web-crawler/tree/main/images/"

  click CRAWLER "https://github.com/afazio1/web-crawler/blob/main/go.mod"
  click CRAWLER "https://github.com/afazio1/web-crawler/blob/main/go.sum"

  %% =======================
  %% Styles
  %% =======================
  classDef process fill:#CFE8FF,stroke:#1E5AA8,stroke-width:1.5px,color:#0B2A4A
  classDef worker fill:#D7F5D7,stroke:#1E7A3D,stroke-width:1.5px,color:#0E3A1E
  classDef queue fill:#FFF2CC,stroke:#B38600,stroke-width:1.5px,color:#4A3700
  classDef datastore fill:#F8D1D1,stroke:#B22222,stroke-width:1.5px,color:#4A0B0B
  classDef index fill:#E7D7FF,stroke:#5B2DA8,stroke-width:1.5px,color:#2B124A
  classDef metrics fill:#D9F0FF,stroke:#0077B6,stroke-width:1.5px,color:#003049
  classDef external fill:#EDEDED,stroke:#666666,stroke-width:1.2px,color:#222222
  classDef note fill:#FFF7E6,stroke:#CC8800,stroke-width:1.2px,color:#5A3B00,stroke-dasharray: 4 3
```
