How mohamedabdellhay/sitemap-generator works
Project type and purpose
• This is a full-stack, single-user, web-based application (“Sitemap Generator Tool”) that crawls an external website, extracts URLs, builds an XML sitemap (optionally gzipped), and streams real-time logs and URL-count updates to the browser.
File structure
• Top-level Python scripts:
• app.py – Flask application entry point, defines routes, SSE endpoint, template rendering, logging setup.
• main.py – (Possibly deprecated or helper) another script entry point.
• sitemap_generator.py – Core crawling and sitemap-building logic (custom module).
• util.py – Helper functions (e.g., URL normalization, XML utilities).
• Configuration files:
• requirements.txt – Python dependencies.
• .gitignore – files/directories to ignore (e.g., logs, last_work).
• Templates directory:
• index.html – UI for form input, SSE log display, URL count, download link.
• sitemaps.html – UI for listing previous runs (last_work).
• last_work/ – Archived sitemap XML (and .gz) outputs from prior runs.
• Logging: app.log (not shown but described in README)
• sitemap_generator module is imported by app.py and util.py.
README for additional insights
• Backend: Python 3.8+, Flask, Python logging, custom sitemap_generator.
• Frontend: HTML + Tailwind CSS + JavaScript + Font Awesome, SSE for real-time logs.
• Deployment: Local Flask server on port 5000.
• Key features: crawling, sitemap generation, gzip option, SSE streaming, dynamic log/URL count, DOM performance constraints.
Main components and interactions
a. Components
1. Browser / Frontend
• index.html UI: form, log area, URL counter, download button.
• sitemaps.html UI: list of past outputs.
• JavaScript: SSE client, DOM updates, form submission via fetch/XHR.
• Tailwind CSS + Font Awesome assets.
2. Flask Backend (app.py)
• HTTP endpoints:
• GET / → serve index.html
• POST /generate → start crawl job
• GET /logs (SSE) → stream Python logging records
• GET /download/<filename> → serve generated .xml or .xml.gz
• GET /sitemaps → serve sitemaps.html + list from last_work/
3. SitemapGenerator Module (sitemap_generator.py)
• Crawl engine: queue of URLs, worker threads (concurrent), delay control, user-agent.
• URL discovery, XML building, gzip compression.
• Emits log records via Python logging.
4. Util Module (util.py)
• URL helpers, file I/O utilities.
5. File Storage
• last_work/ directory: stores generated sitemap files.
• app.log: persistent log file.
6. External Website (target)
• HTTP(S) host being crawled.
7. Dependencies
• Flask framework, Python logging, custom module “sitemap_generator”.
b. Relationships / Data Flow
1. User → Browser fills form → Browser POST /generate
2. Flask /generate route invokes SitemapGenerator
3. SitemapGenerator fetches pages from external site, discovers URLs
4. SitemapGenerator writes to XML, optionally gzips, saves to last_work/
5. Python logging in both app.py and sitemap_generator emits log events
6. SSE endpoint streams log events to browser
7. Browser displays log lines and updates URL count
8. Upon completion, Browser GET /download/<filename> to retrieve sitemap
c. Architectural patterns / design principles
• Layered architecture: Presentation (HTML/JS/CSS) ↔ Application (Flask controllers) ↔ Domain (crawler module) ↔ Infrastructure (file I/O, logging).
• Producer/consumer within the crawler (queue, worker threads).
• Event streaming (SSE) for asynchronous log updates.
• Single-process, multi-threaded work model.
d. Key technologies / frameworks
• Python 3.8+, Flask, threading or asyncio within sitemap_generator
• Python logging module
• HTML5, Tailwind CSS, JavaScript EventSource (SSE)
Detail
• Concurrent workers as internal detail of crawler.
• Configuration settings flow from UI into the crawler.
• That logs are limited to 100 entries in the DOM (performance note).