How phialsbasement/librecrawl works
LibreCrawl is a multi-user web crawler for SEO auditing. A user submits a site or URL list through the web application; Flask routes configure and start a crawler, which fetches pages, extracts SEO and link data, and produces issue, site-wide, and n-gram analyses. Results can be monitored, revisited, compared, and exported. Authentication, per-user settings, and crawl persistence support that workflow. The README documents additional UI, export, and PageSpeed capabilities. Some links are omitted where the sampled excerpts do not show the route or call.
Main entry points: crawler.py Crawler orchestrator · main.py Flask routes · app.js Web interface · zoho_oauth.py Zoho sign-in
Components 23
Web application
- Web interface
web/static/js/app.js - Web views
web/templates/dashboard.html - Flask routes
main.py - Result exports
main.py
Crawling
- Crawler orchestrator
src/crawler.py - SEO extraction
src/core/seo_extractor.py - Link management
src/core/link_manager.py - Sitemap discovery
src/core/sitemap_parser.py - JavaScript rendering
src/core/js_renderer.py - Custom extraction
src/core/custom_extractor.py
SEO analysis
- Issue detection
src/core/issue_detector.py - Crawl analysis
src/core/crawl_analysis.py - Crawl comparison
src/core/crawl_compare.py - N-gram analysis
src/core/ngram_analyzer.py
Accounts and data
- Authentication
src/auth_db.py - Crawl history
src/crawl_db.py - User settings
src/settings_manager.py
Integrations
- Zoho sign-in
src/zoho_oauth.py - Verification email
src/email_service.py
Other components
- Crawl user
- Crawled websites
- PageSpeed Insights
- SMTP server
Connections 24
- Crawl user → Web interface: uses
README.md - Web interface → Web views: renders
- Web interface → Flask routes: submits crawl
- Flask routes → Crawler orchestrator: starts crawl
main.py - Flask routes → User settings: configures
main.py - Flask routes → Authentication: authenticates
src/auth_db.py - Flask routes → Zoho sign-in: handles callback
main.py - Flask routes → Verification email: sends verification
main.py - Flask routes → Crawl comparison: compares crawls
main.py - Flask routes → N-gram analysis: computes phrases
main.py - Flask routes → Crawl history: deletes crawl
main.py - Crawler orchestrator → Crawled websites: fetches pages
README.md - Crawler orchestrator → SEO extraction: extracts page data
src/crawler.py - Crawler orchestrator → Link management: manages links
src/crawler.py - Crawler orchestrator → Sitemap discovery: discovers sitemaps
src/crawler.py - Crawler orchestrator → JavaScript rendering: renders JavaScript
src/crawler.py - Crawler orchestrator → Custom extraction: runs custom rules
src/crawler.py - Crawler orchestrator → Issue detection: detects issues
src/crawler.py - Crawler orchestrator → Crawl analysis: analyzes completed crawl
src/crawler.py - Crawler orchestrator → Crawl history: persists crawl
src/crawler.py - Crawler orchestrator → PageSpeed Insights: requests metrics
README.md - Flask routes → Result exports: exports results
README.md - Verification email → SMTP server: sends mail
src/email_service.py - Zoho sign-in → Crawl user: returns identity
src/zoho_oauth.py