# siddheshpohare/distributed-web-crawler-search-engine architecture

> Architecture diagram of the GitHub repository siddheshpohare/distributed-web-crawler-search-engine, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-03). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/siddheshpohare/distributed-web-crawler-search-engine
- Repository: https://github.com/siddheshpohare/distributed-web-crawler-search-engine

## Overview

This repository currently implements a synchronous web crawler and a FastAPI/PostgreSQL service around it. The main user workflow is to submit seed URLs and a page limit, fetch pages, extract links, schedule discovered URLs fairly across domains, save crawl outcomes, and query pages or crawl status. A separate command-line entry point runs the crawler without the API. The README also describes Redis-based distributed crawling, indexing/search, and deployment/monitoring as future phases; those are not implemented in the sampled runtime, so they are not drawn as active components.

Main entry points: [`api/routes/crawl.py`](https://github.com/siddheshpohare/distributed-web-crawler-search-engine/blob/main/api/routes/crawl.py) (Crawl control), [`api/main.py`](https://github.com/siddheshpohare/distributed-web-crawler-search-engine/blob/main/api/main.py) (FastAPI app), [`api/routes/pages.py`](https://github.com/siddheshpohare/distributed-web-crawler-search-engine/blob/main/api/routes/pages.py) (Page queries), [`crawler/fetcher.py`](https://github.com/siddheshpohare/distributed-web-crawler-search-engine/blob/main/crawler/fetcher.py) (HTTP fetcher)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_api["API service"]
  node_app["FastAPI app<br/>[main.py]"]
  node_health["Health route<br/>[health.py]"]
  node_crawl_route["Crawl control<br/>[crawl.py]"]
  node_pages_route["Page queries<br/>[pages.py]"]
end

subgraph group_crawl["Crawl runtime"]
  node_cli_loop["CLI crawl loop<br/>[main.py]"]
  node_frontier["URL frontier<br/>[frontier.py]"]
  node_fetcher["HTTP fetcher<br/>[fetcher.py]"]
  node_parser["Link parser<br/>[parser.py]"]
end

subgraph group_storage["Persistence"]
  node_session["DB sessions<br/>[session.py]"]
  node_page_model["Page model<br/>[models.py]"]
  node_postgres[("PostgreSQL")]
end

subgraph group_contracts["API contracts"]
  node_schemas["Request responses<br/>[schemas.py]"]
end

node_client(("API client"))
node_web(("Web pages"))

node_client -->|"sends requests"| node_app
node_app -->|"mounts router"| node_health
node_app -->|"mounts router"| node_crawl_route
node_app -->|"mounts router"| node_pages_route
node_client -->|"starts crawl"| node_crawl_route
node_client -->|"queries pages"| node_pages_route
node_crawl_route -->|"validates and shapes"| node_schemas
node_pages_route -->|"shapes responses"| node_schemas
node_crawl_route -->|"schedules URLs"| node_frontier
node_crawl_route -->|"fetches pages"| node_fetcher
node_crawl_route -->|"extracts links"| node_parser
node_crawl_route -->|"gets DB session"| node_session
node_crawl_route -->|"upserts outcomes"| node_page_model
node_pages_route -->|"gets DB session"| node_session
node_pages_route -->|"queries pages"| node_page_model
node_session -->|"opens connections"| node_postgres
node_page_model -->|"uses ORM base"| node_session
node_app -->|"creates tables"| node_session
node_web -->|"serves HTML"| node_fetcher
node_cli_loop -->|"schedules URLs"| node_frontier
node_cli_loop -->|"fetches pages"| node_fetcher
node_cli_loop -->|"extracts links"| node_parser
node_parser -->|"returns discovered URLs"| node_frontier
node_fetcher -->|"returns HTML"| node_cli_loop
node_fetcher -->|"returns HTML"| node_crawl_route

click node_app "https://github.com/siddheshpohare/distributed-web-crawler-search-engine/blob/main/api/main.py"
click node_health "https://github.com/siddheshpohare/distributed-web-crawler-search-engine/blob/main/api/routes/health.py"
click node_crawl_route "https://github.com/siddheshpohare/distributed-web-crawler-search-engine/blob/main/api/routes/crawl.py"
click node_pages_route "https://github.com/siddheshpohare/distributed-web-crawler-search-engine/blob/main/api/routes/pages.py"
click node_cli_loop "https://github.com/siddheshpohare/distributed-web-crawler-search-engine/blob/main/crawler/main.py"
click node_frontier "https://github.com/siddheshpohare/distributed-web-crawler-search-engine/blob/main/crawler/frontier.py"
click node_fetcher "https://github.com/siddheshpohare/distributed-web-crawler-search-engine/blob/main/crawler/fetcher.py"
click node_parser "https://github.com/siddheshpohare/distributed-web-crawler-search-engine/blob/main/crawler/parser.py"
click node_schemas "https://github.com/siddheshpohare/distributed-web-crawler-search-engine/blob/main/api/schemas.py"
click node_session "https://github.com/siddheshpohare/distributed-web-crawler-search-engine/blob/main/api/db/session.py"
click node_page_model "https://github.com/siddheshpohare/distributed-web-crawler-search-engine/blob/main/api/db/models.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_app,node_health,node_crawl_route,node_pages_route,node_client toneBlue
class node_cli_loop,node_frontier,node_fetcher,node_parser toneAmber
class node_session,node_page_model,node_postgres toneMint
class node_schemas toneRose
class node_web toneIndigo
```

## Components

### API service

- [FastAPI app](https://github.com/siddheshpohare/distributed-web-crawler-search-engine/blob/main/api/main.py): `api/main.py`
- [Health route](https://github.com/siddheshpohare/distributed-web-crawler-search-engine/blob/main/api/routes/health.py): `api/routes/health.py`
- [Crawl control](https://github.com/siddheshpohare/distributed-web-crawler-search-engine/blob/main/api/routes/crawl.py): `api/routes/crawl.py`
- [Page queries](https://github.com/siddheshpohare/distributed-web-crawler-search-engine/blob/main/api/routes/pages.py): `api/routes/pages.py`

### Crawl runtime

- [CLI crawl loop](https://github.com/siddheshpohare/distributed-web-crawler-search-engine/blob/main/crawler/main.py): `crawler/main.py`
- [URL frontier](https://github.com/siddheshpohare/distributed-web-crawler-search-engine/blob/main/crawler/frontier.py): `crawler/frontier.py`
- [HTTP fetcher](https://github.com/siddheshpohare/distributed-web-crawler-search-engine/blob/main/crawler/fetcher.py): `crawler/fetcher.py`
- [Link parser](https://github.com/siddheshpohare/distributed-web-crawler-search-engine/blob/main/crawler/parser.py): `crawler/parser.py`

### Persistence

- [DB sessions](https://github.com/siddheshpohare/distributed-web-crawler-search-engine/blob/main/api/db/session.py): `api/db/session.py`
- [Page model](https://github.com/siddheshpohare/distributed-web-crawler-search-engine/blob/main/api/db/models.py): `api/db/models.py`
- **PostgreSQL**

### API contracts

- [Request responses](https://github.com/siddheshpohare/distributed-web-crawler-search-engine/blob/main/api/schemas.py): `api/schemas.py`

### Other components

- **API client**
- **Web pages**

## Connections

- API client → FastAPI app: sends requests (evidence: `README.md`)
- FastAPI app → Health route: mounts router (evidence: `api/main.py`)
- FastAPI app → Crawl control: mounts router (evidence: `api/main.py`)
- FastAPI app → Page queries: mounts router (evidence: `api/main.py`)
- API client → Crawl control: starts crawl (evidence: `README.md`)
- API client → Page queries: queries pages (evidence: `README.md`)
- Crawl control → Request responses: validates and shapes (evidence: `api/routes/crawl.py`)
- Page queries → Request responses: shapes responses (evidence: `api/routes/pages.py`)
- Crawl control → URL frontier: schedules URLs (evidence: `api/routes/crawl.py`)
- Crawl control → HTTP fetcher: fetches pages (evidence: `api/routes/crawl.py`)
- Crawl control → Link parser: extracts links (evidence: `api/routes/crawl.py`)
- Crawl control → DB sessions: gets DB session (evidence: `api/routes/crawl.py`)
- Crawl control → Page model: upserts outcomes (evidence: `api/routes/crawl.py`)
- Page queries → DB sessions: gets DB session (evidence: `api/routes/pages.py`)
- Page queries → Page model: queries pages (evidence: `api/routes/pages.py`)
- DB sessions → PostgreSQL: opens connections (evidence: `api/db/session.py`)
- Page model → DB sessions: uses ORM base (evidence: `api/db/models.py`)
- FastAPI app → DB sessions: creates tables (evidence: `api/main.py`)
- Web pages → HTTP fetcher: serves HTML (evidence: `crawler/fetcher.py`)
- CLI crawl loop → URL frontier: schedules URLs (evidence: `crawler/main.py`)
- CLI crawl loop → HTTP fetcher: fetches pages (evidence: `crawler/main.py`)
- CLI crawl loop → Link parser: extracts links (evidence: `crawler/main.py`)
- Link parser → URL frontier: returns discovered URLs (evidence: `crawler/main.py`)
- HTTP fetcher → CLI crawl loop: returns HTML (evidence: `crawler/main.py`)
- HTTP fetcher → Crawl control: returns HTML (evidence: `api/routes/crawl.py`)
