How siddheshpohare/distributed-web-crawler-search-engine works
This repository currently implements a synchronous web crawler and a FastAPI/PostgreSQL service around it. The main user workflow is to submit seed URLs and a page limit, fetch pages, extract links, schedule discovered URLs fairly across domains, save crawl outcomes, and query pages or crawl status. A separate command-line entry point runs the crawler without the API. The README also describes Redis-based distributed crawling, indexing/search, and deployment/monitoring as future phases; those are not implemented in the sampled runtime, so they are not drawn as active components.
Main entry points: crawl.py Crawl control · main.py FastAPI app · pages.py Page queries · fetcher.py HTTP fetcher
Components 14
API service
- FastAPI app
api/main.py - Health route
api/routes/health.py - Crawl control
api/routes/crawl.py - Page queries
api/routes/pages.py
Crawl runtime
- CLI crawl loop
crawler/main.py - URL frontier
crawler/frontier.py - HTTP fetcher
crawler/fetcher.py - Link parser
crawler/parser.py
Persistence
- DB sessions
api/db/session.py - Page model
api/db/models.py - PostgreSQL
API contracts
- Request responses
api/schemas.py
Other components
- API client
- Web pages
Connections 25
- API client → FastAPI app: sends requests
README.md - FastAPI app → Health route: mounts router
api/main.py - FastAPI app → Crawl control: mounts router
api/main.py - FastAPI app → Page queries: mounts router
api/main.py - API client → Crawl control: starts crawl
README.md - API client → Page queries: queries pages
README.md - Crawl control → Request responses: validates and shapes
api/routes/crawl.py - Page queries → Request responses: shapes responses
api/routes/pages.py - Crawl control → URL frontier: schedules URLs
api/routes/crawl.py - Crawl control → HTTP fetcher: fetches pages
api/routes/crawl.py - Crawl control → Link parser: extracts links
api/routes/crawl.py - Crawl control → DB sessions: gets DB session
api/routes/crawl.py - Crawl control → Page model: upserts outcomes
api/routes/crawl.py - Page queries → DB sessions: gets DB session
api/routes/pages.py - Page queries → Page model: queries pages
api/routes/pages.py - DB sessions → PostgreSQL: opens connections
api/db/session.py - Page model → DB sessions: uses ORM base
api/db/models.py - FastAPI app → DB sessions: creates tables
api/main.py - Web pages → HTTP fetcher: serves HTML
crawler/fetcher.py - CLI crawl loop → URL frontier: schedules URLs
crawler/main.py - CLI crawl loop → HTTP fetcher: fetches pages
crawler/main.py - CLI crawl loop → Link parser: extracts links
crawler/main.py - Link parser → URL frontier: returns discovered URLs
crawler/main.py - HTTP fetcher → CLI crawl loop: returns HTML
crawler/main.py - HTTP fetcher → Crawl control: returns HTML
api/routes/crawl.py