# mohamedabdellhay/sitemap-generator architecture

> Architecture diagram of the GitHub repository mohamedabdellhay/sitemap-generator, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-12). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/mohamedabdellhay/sitemap-generator
- Repository: https://github.com/mohamedabdellhay/sitemap-generator

## Overview

##### Project type and purpose
- This is a full-stack, single-user, web-based application (“Sitemap Generator Tool”) that crawls an external website, extracts URLs, builds an XML sitemap (optionally gzipped), and streams real-time logs and URL-count updates to the browser.

##### File structure
- Top-level Python scripts:
- app.py – Flask application entry point, defines routes, SSE endpoint, template rendering, logging setup.
- main.py – (Possibly deprecated or helper) another script entry point.
- sitemap_generator.py – Core crawling and sitemap-building logic (custom module).
- util.py – Helper functions (e.g., URL normalization, XML utilities).
- Configuration files:
- requirements.txt – Python dependencies.
- .gitignore – files/directories to ignore (e.g., logs, last_work).
- Templates directory:
- index.html – UI for form input, SSE log display, URL count, download link.
- sitemaps.html – UI for listing previous runs (last_work).
- last_work/ – Archived sitemap XML (and .gz) outputs from prior runs.
- Logging: app.log (not shown but described in README)
- sitemap_generator module is imported by app.py and util.py.

##### README for additional insights
- Backend: Python 3.8+, Flask, Python logging, custom sitemap_generator.
- Frontend: HTML + Tailwind CSS + JavaScript + Font Awesome, SSE for real-time logs.
- Deployment: Local Flask server on port 5000.
- Key features: crawling, sitemap generation, gzip option, SSE streaming, dynamic log/URL count, DOM performance constraints.

##### Main components and interactions
a. Components
1. Browser / Frontend
- index.html UI: form, log area, URL counter, download button.
- sitemaps.html UI: list of past outputs.
- JavaScript: SSE client, DOM updates, form submission via fetch/XHR.
- Tailwind CSS + Font Awesome assets.
2. Flask Backend (app.py)
- HTTP endpoints:
- GET / → serve index.html
- POST /generate → start crawl job
- GET /logs (SSE) → stream Python logging records
- GET /download/<filename> → serve generated .xml or .xml.gz
- GET /sitemaps → serve sitemaps.html + list from last_work/
3. SitemapGenerator Module (sitemap_generator.py)
- Crawl engine: queue of URLs, worker threads (concurrent), delay control, user-agent.
- URL discovery, XML building, gzip compression.
- Emits log records via Python logging.
4. Util Module (util.py)
- URL helpers, file I/O utilities.
5. File Storage
- last_work/ directory: stores generated sitemap files.
- app.log: persistent log file.
6. External Website (target)
- HTTP(S) host being crawled.
7. Dependencies
- Flask framework, Python logging, custom module “sitemap_generator”.

b. Relationships / Data Flow
1. User → Browser fills form → Browser POST /generate
2. Flask /generate route invokes SitemapGenerator
3. SitemapGenerator fetches pages from external site, discovers URLs
4. SitemapGenerator writes to XML, optionally gzips, saves to last_work/
5. Python logging in both app.py and sitemap_generator emits log events
6. SSE endpoint streams log events to browser
7. Browser displays log lines and updates URL count
8. Upon completion, Browser GET /download/<filename> to retrieve sitemap

c. Architectural patterns / design principles
- Layered architecture: Presentation (HTML/JS/CSS) ↔ Application (Flask controllers) ↔ Domain (crawler module) ↔ Infrastructure (file I/O, logging).
- Producer/consumer within the crawler (queue, worker threads).
- Event streaming (SSE) for asynchronous log updates.
- Single-process, multi-threaded work model.

d. Key technologies / frameworks
- Python 3.8+, Flask, threading or asyncio within sitemap_generator
- Python logging module
- HTML5, Tailwind CSS, JavaScript EventSource (SSE)

##### Detail
- Concurrent workers as internal detail of crawler.
- Configuration settings flow from UI into the crawler.
- That logs are limited to 100 entries in the DOM (performance note).

## Diagram (Mermaid)

```mermaid
flowchart TD
    subgraph "Frontend (Browser)"
        idx["index.html UI: Form, Logs, URL Count, Download"]:::frontend
        sitem["sitemaps.html UI: Past Runs Listing"]:::frontend
        js["JavaScript (EventSource SSE)\nTailwind CSS\nFont Awesome"]:::frontend
    end

    subgraph "Backend (Flask)"
        app["Flask App (app.py):\nRoutes, SSE Endpoint, Logging Setup"]:::backend
        main["main.py (helper entry point)"]:::backend
        util["util.py: URL Helpers & File I/O"]:::backend
    end

    subgraph "Domain (Crawler)"
        sg["Crawler Engine (sitemap_generator.py):\nQueue, Workers, XML & Gzip"]:::domain
    end

    storage[("File Storage\nlast_work/ (XML & .gz)\napp.log")]:::storage
    cfg["requirements.txt\n.gitignore"]:::backend
    doc["README.md"]:::backend
    ext(("External Website")):::external

    %% Data Flows
    idx -->|POST /generate| app
    sitem -->|GET /sitemaps| app
    idx -->|EventSource /logs| app
    app -->|serve UI| idx
    app -->|serve UI| sitem
    app -->|invokes| sg
    sg -->|HTTP GET pages| ext
    sg -->|save XML & gz| storage
    app -->|serve download| storage
    app -->|logging| storage
    sg -->|logging| storage
    app -->|GET /download| idx
    app -->|uses| util
    sg -->|uses| util

    %% Assets interaction
    idx --> js
    sitem --> js

    %% Config & Docs
    app --> cfg
    sg --> cfg
    main --> cfg
    app --> doc
    sg --> doc
    util --> doc

    %% Click Events
    click idx "https://github.com/mohamedabdellhay/sitemap-generator/blob/main/templates/index.html"
    click sitem "https://github.com/mohamedabdellhay/sitemap-generator/blob/main/templates/sitemaps.html"
    click app "https://github.com/mohamedabdellhay/sitemap-generator/blob/main/app.py"
    click main "https://github.com/mohamedabdellhay/sitemap-generator/blob/main/main.py"
    click sg "https://github.com/mohamedabdellhay/sitemap-generator/blob/main/sitemap_generator.py"
    click util "https://github.com/mohamedabdellhay/sitemap-generator/blob/main/util.py"
    click storage "https://github.com/mohamedabdellhay/sitemap-generator/tree/main/last_work/"
    click cfg "https://github.com/mohamedabdellhay/sitemap-generator/blob/main/requirements.txt"
    click cfg "https://github.com/mohamedabdellhay/sitemap-generator/blob/main/.gitignore"
    click doc "https://github.com/mohamedabdellhay/sitemap-generator/blob/main/README.md"

    %% Styles
    classDef frontend fill:#cce5ff,stroke:#1f78b4,stroke-width:2px
    classDef backend fill:#d4edda,stroke:#28a745,stroke-width:2px
    classDef domain fill:#fff3cd,stroke:#ffc107,stroke-width:2px
    classDef storage fill:#f8d7da,stroke:#dc3545,stroke-width:2px
    classDef external fill:#e2e3e5,stroke:#6c757d,stroke-width:2px,stroke-dasharray: 5 5
```
