# phialsbasement/librecrawl architecture

> Architecture diagram of the GitHub repository phialsbasement/librecrawl, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-16). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/phialsbasement/librecrawl
- Repository: https://github.com/phialsbasement/librecrawl

## Overview

LibreCrawl is a multi-user web crawler for SEO auditing. A user submits a site or URL list through the web application; Flask routes configure and start a crawler, which fetches pages, extracts SEO and link data, and produces issue, site-wide, and n-gram analyses. Results can be monitored, revisited, compared, and exported. Authentication, per-user settings, and crawl persistence support that workflow. The README documents additional UI, export, and PageSpeed capabilities. Some links are omitted where the sampled excerpts do not show the route or call.

Main entry points: [`src/crawler.py`](https://github.com/phialsbasement/librecrawl/blob/main/src/crawler.py) (Crawler orchestrator), [`main.py`](https://github.com/phialsbasement/librecrawl/blob/main/main.py) (Flask routes), [`web/static/js/app.js`](https://github.com/phialsbasement/librecrawl/blob/main/web/static/js/app.js) (Web interface), [`src/zoho_oauth.py`](https://github.com/phialsbasement/librecrawl/blob/main/src/zoho_oauth.py) (Zoho sign-in)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_web["Web application"]
  node_browser["Web interface<br/>[app.js]"]
  node_views["Web views<br/>[dashboard.html]"]
  node_server["Flask routes<br/>[main.py]"]
  node_exports["Result exports<br/>[main.py]"]
end

subgraph group_crawl["Crawling"]
  node_crawler["Crawler orchestrator<br/>[crawler.py]"]
  node_seo["SEO extraction<br/>[seo_extractor.py]"]
  node_links["Link management<br/>[link_manager.py]"]
  node_sitemaps["Sitemap discovery<br/>[sitemap_parser.py]"]
  node_render["JavaScript rendering<br/>[js_renderer.py]"]
  node_extractors["Custom extraction"]
end

subgraph group_analysis["SEO analysis"]
  node_issues["Issue detection<br/>[issue_detector.py]"]
  node_siteanalysis["Crawl analysis<br/>[crawl_analysis.py]"]
  node_compare["Crawl comparison<br/>[crawl_compare.py]"]
  node_ngrams["N-gram analysis<br/>[ngram_analyzer.py]"]
end

subgraph group_state["Accounts and data"]
  node_accounts[("Authentication<br/>[auth_db.py]")]
  node_crawlstore[("Crawl history<br/>[crawl_db.py]")]
  node_settings["User settings"]
end

subgraph group_integrations["Integrations"]
  node_zoho["Zoho sign-in<br/>[zoho_oauth.py]"]
  node_email["Verification email<br/>[email_service.py]"]
end

node_user(("Crawl user"))
node_target(("Crawled websites"))
node_pagespeed(("PageSpeed Insights"))
node_smtp(("SMTP server"))

node_user -->|"uses"| node_browser
node_browser -.->|"renders"| node_views
node_browser -.->|"submits crawl"| node_server
node_server -->|"starts crawl"| node_crawler
node_server -->|"configures"| node_settings
node_server -->|"authenticates"| node_accounts
node_server -->|"handles callback"| node_zoho
node_server -.->|"sends verification"| node_email
node_server -->|"compares crawls"| node_compare
node_server -->|"computes phrases"| node_ngrams
node_server -->|"deletes crawl"| node_crawlstore
node_crawler -->|"fetches pages"| node_target
node_crawler -->|"extracts page data"| node_seo
node_crawler -->|"manages links"| node_links
node_crawler -->|"discovers sitemaps"| node_sitemaps
node_crawler -->|"renders JavaScript"| node_render
node_crawler -->|"runs custom rules"| node_extractors
node_crawler -->|"detects issues"| node_issues
node_crawler -->|"analyzes completed crawl"| node_siteanalysis
node_crawler -->|"persists crawl"| node_crawlstore
node_crawler -.->|"requests metrics"| node_pagespeed
node_server -->|"exports results"| node_exports
node_email -->|"sends mail"| node_smtp
node_zoho -->|"returns identity"| node_user

click node_browser "https://github.com/phialsbasement/librecrawl/blob/main/web/static/js/app.js"
click node_views "https://github.com/phialsbasement/librecrawl/blob/main/web/templates/dashboard.html"
click node_server "https://github.com/phialsbasement/librecrawl/blob/main/main.py"
click node_exports "https://github.com/phialsbasement/librecrawl/blob/main/main.py"
click node_crawler "https://github.com/phialsbasement/librecrawl/blob/main/src/crawler.py"
click node_seo "https://github.com/phialsbasement/librecrawl/blob/main/src/core/seo_extractor.py"
click node_links "https://github.com/phialsbasement/librecrawl/blob/main/src/core/link_manager.py"
click node_sitemaps "https://github.com/phialsbasement/librecrawl/blob/main/src/core/sitemap_parser.py"
click node_render "https://github.com/phialsbasement/librecrawl/blob/main/src/core/js_renderer.py"
click node_extractors "https://github.com/phialsbasement/librecrawl/blob/main/src/core/custom_extractor.py"
click node_issues "https://github.com/phialsbasement/librecrawl/blob/main/src/core/issue_detector.py"
click node_siteanalysis "https://github.com/phialsbasement/librecrawl/blob/main/src/core/crawl_analysis.py"
click node_compare "https://github.com/phialsbasement/librecrawl/blob/main/src/core/crawl_compare.py"
click node_ngrams "https://github.com/phialsbasement/librecrawl/blob/main/src/core/ngram_analyzer.py"
click node_accounts "https://github.com/phialsbasement/librecrawl/blob/main/src/auth_db.py"
click node_crawlstore "https://github.com/phialsbasement/librecrawl/blob/main/src/crawl_db.py"
click node_settings "https://github.com/phialsbasement/librecrawl/blob/main/src/settings_manager.py"
click node_zoho "https://github.com/phialsbasement/librecrawl/blob/main/src/zoho_oauth.py"
click node_email "https://github.com/phialsbasement/librecrawl/blob/main/src/email_service.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_browser,node_views,node_server,node_exports,node_user toneBlue
class node_crawler,node_seo,node_links,node_sitemaps,node_render,node_extractors toneAmber
class node_issues,node_siteanalysis,node_compare,node_ngrams,node_smtp toneMint
class node_accounts,node_crawlstore,node_settings toneRose
class node_zoho,node_email,node_target,node_pagespeed toneIndigo
```

## Components

### Web application

- [Web interface](https://github.com/phialsbasement/librecrawl/blob/main/web/static/js/app.js): `web/static/js/app.js`
- [Web views](https://github.com/phialsbasement/librecrawl/blob/main/web/templates/dashboard.html): `web/templates/dashboard.html`
- [Flask routes](https://github.com/phialsbasement/librecrawl/blob/main/main.py): `main.py`
- [Result exports](https://github.com/phialsbasement/librecrawl/blob/main/main.py): `main.py`

### Crawling

- [Crawler orchestrator](https://github.com/phialsbasement/librecrawl/blob/main/src/crawler.py): `src/crawler.py`
- [SEO extraction](https://github.com/phialsbasement/librecrawl/blob/main/src/core/seo_extractor.py): `src/core/seo_extractor.py`
- [Link management](https://github.com/phialsbasement/librecrawl/blob/main/src/core/link_manager.py): `src/core/link_manager.py`
- [Sitemap discovery](https://github.com/phialsbasement/librecrawl/blob/main/src/core/sitemap_parser.py): `src/core/sitemap_parser.py`
- [JavaScript rendering](https://github.com/phialsbasement/librecrawl/blob/main/src/core/js_renderer.py): `src/core/js_renderer.py`
- [Custom extraction](https://github.com/phialsbasement/librecrawl/blob/main/src/core/custom_extractor.py): `src/core/custom_extractor.py`

### SEO analysis

- [Issue detection](https://github.com/phialsbasement/librecrawl/blob/main/src/core/issue_detector.py): `src/core/issue_detector.py`
- [Crawl analysis](https://github.com/phialsbasement/librecrawl/blob/main/src/core/crawl_analysis.py): `src/core/crawl_analysis.py`
- [Crawl comparison](https://github.com/phialsbasement/librecrawl/blob/main/src/core/crawl_compare.py): `src/core/crawl_compare.py`
- [N-gram analysis](https://github.com/phialsbasement/librecrawl/blob/main/src/core/ngram_analyzer.py): `src/core/ngram_analyzer.py`

### Accounts and data

- [Authentication](https://github.com/phialsbasement/librecrawl/blob/main/src/auth_db.py): `src/auth_db.py`
- [Crawl history](https://github.com/phialsbasement/librecrawl/blob/main/src/crawl_db.py): `src/crawl_db.py`
- [User settings](https://github.com/phialsbasement/librecrawl/blob/main/src/settings_manager.py): `src/settings_manager.py`

### Integrations

- [Zoho sign-in](https://github.com/phialsbasement/librecrawl/blob/main/src/zoho_oauth.py): `src/zoho_oauth.py`
- [Verification email](https://github.com/phialsbasement/librecrawl/blob/main/src/email_service.py): `src/email_service.py`

### Other components

- **Crawl user**
- **Crawled websites**
- **PageSpeed Insights**
- **SMTP server**

## Connections

- Crawl user → Web interface: uses (evidence: `README.md`)
- Web interface → Web views: renders
- Web interface → Flask routes: submits crawl
- Flask routes → Crawler orchestrator: starts crawl (evidence: `main.py`)
- Flask routes → User settings: configures (evidence: `main.py`)
- Flask routes → Authentication: authenticates (evidence: `src/auth_db.py`)
- Flask routes → Zoho sign-in: handles callback (evidence: `main.py`)
- Flask routes → Verification email: sends verification (evidence: `main.py`)
- Flask routes → Crawl comparison: compares crawls (evidence: `main.py`)
- Flask routes → N-gram analysis: computes phrases (evidence: `main.py`)
- Flask routes → Crawl history: deletes crawl (evidence: `main.py`)
- Crawler orchestrator → Crawled websites: fetches pages (evidence: `README.md`)
- Crawler orchestrator → SEO extraction: extracts page data (evidence: `src/crawler.py`)
- Crawler orchestrator → Link management: manages links (evidence: `src/crawler.py`)
- Crawler orchestrator → Sitemap discovery: discovers sitemaps (evidence: `src/crawler.py`)
- Crawler orchestrator → JavaScript rendering: renders JavaScript (evidence: `src/crawler.py`)
- Crawler orchestrator → Custom extraction: runs custom rules (evidence: `src/crawler.py`)
- Crawler orchestrator → Issue detection: detects issues (evidence: `src/crawler.py`)
- Crawler orchestrator → Crawl analysis: analyzes completed crawl (evidence: `src/crawler.py`)
- Crawler orchestrator → Crawl history: persists crawl (evidence: `src/crawler.py`)
- Crawler orchestrator → PageSpeed Insights: requests metrics (evidence: `README.md`)
- Flask routes → Result exports: exports results (evidence: `README.md`)
- Verification email → SMTP server: sends mail (evidence: `src/email_service.py`)
- Zoho sign-in → Crawl user: returns identity (evidence: `src/zoho_oauth.py`)
