# sirioberati/webtwin architecture

> Architecture diagram of the GitHub repository sirioberati/webtwin, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-12). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/sirioberati/webtwin
- Repository: https://github.com/sirioberati/webtwin

## Overview

##### Project Type and Purpose
- This is a Python-based full-stack web application (Flask) whose sole purpose is to provide an HTTP interface for extracting and packaging entire websites into a downloadable ZIP archive.
- The core “business logic” is an asset-extraction pipeline that can run in two modes: simple HTTP fetch (requests) or advanced JavaScript rendering (Selenium + Chrome WebDriver).

##### File Structure Analysis
Top-level files:
- app.py – Flask application entrypoint and HTTP routes
- setup.py / requirements.txt – packaging and dependencies
- app_architecture.md + docs/app_architecture_overview.md – textual and image descriptions of architecture
- templates/ – Jinja2 HTML templates for the UI (index.html: input form; components.html: detailed UI)
- LICENSE, CONTRIBUTING.md – repo governance

No database; no microservices. The architecture is a monolithic Flask service layered with template rendering and a modular extraction core.

##### README and Architecture Insights
- The README already spells out a multi-stage pipeline: URL submission → (optional) JS rendering → HTML parsing → asset discovery → parallel downloading → ZIP packaging → file download response.
- Key technologies: Flask (web server), requests (HTTP client), Selenium WebDriver (JS rendering), BeautifulSoup (HTML parsing), Python’s zipfile (ZIP creator).

##### Essential Details to Capture
- The bifurcation in the pipeline for advanced rendering (Selenium) vs. simple HTTP fetch.
- The parallel/concurrent nature of asset downloading.
- The packaging step and final HTTP response back to the client.
- The fact that there is no permanent storage–everything is generated on the fly and streamed.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Frontend Layer
    subgraph "Frontend Layer"
        Browser["Web Browser"]:::ui
        Index["templates/index.html"]:::ui
        Components["templates/components.html"]:::ui
    end

    %% Backend Layer
    subgraph "Backend Layer"
        Flask["Flask Web Server"]:::server
        Controller["Controller / Routes"]:::server
    end

    %% Extraction Core
    subgraph "Extraction Core"
        Decision{"Rendering Mode?"}:::core
        HttpClient["HTTP Client Module\n(requests)"]:::core
        Selenium["Selenium Renderer Module\n(WebDriver)"]:::core
        Parser["Content Parser Module\n(BeautifulSoup)"]:::core
        Downloader["Asset Downloader Module\n(Concurrent Fetch)"]:::core
        Zipper["ZIP Creator Module\n(zipfile)"]:::core
    end

    %% External Systems
    subgraph "External Systems"
        WWW["Target Website"]:::external
        WebDriver["Chrome WebDriver"]:::external
    end

    %% Flow
    Browser -->|"HTTP Request"| Flask
    Flask -->|"URL"| Controller
    Controller --> Decision
    Decision -->|"Simple Fetch"| HttpClient
    Decision -->|"Advanced Render"| Selenium
    HttpClient -->|"HTML Content"| Parser
    Selenium -->|"Rendered HTML"| Parser
    Parser -->|"Asset URLs"| Downloader
    Downloader -->|"Assets"| Zipper
    Zipper -->|"ZIP File"| Controller
    Controller -->|"HTTP ZIP Response"| Browser

    %% Click Events
    click Flask "https://github.com/sirioberati/webtwin/blob/main/app.py"
    click Controller "https://github.com/sirioberati/webtwin/blob/main/app.py"
    click HttpClient "https://github.com/sirioberati/webtwin/blob/main/app.py"
    click Selenium "https://github.com/sirioberati/webtwin/blob/main/app.py"
    click Parser "https://github.com/sirioberati/webtwin/blob/main/app.py"
    click Downloader "https://github.com/sirioberati/webtwin/blob/main/app.py"
    click Zipper "https://github.com/sirioberati/webtwin/blob/main/app.py"
    click Index "https://github.com/sirioberati/webtwin/blob/main/templates/index.html"
    click Components "https://github.com/sirioberati/webtwin/blob/main/templates/components.html"
    click requirements "https://github.com/sirioberati/webtwin/blob/main/requirements.txt"
    click setup "https://github.com/sirioberati/webtwin/blob/main/setup.py"
    click ArchDoc1 "https://github.com/sirioberati/webtwin/blob/main/app_architecture.md"
    click ArchDoc2 "https://github.com/sirioberati/webtwin/blob/main/docs/app_architecture_overview.md"
    click Contrib "https://github.com/sirioberati/webtwin/blob/main/CONTRIBUTING.md"
    click License "https://github.com/sirioberati/webtwin/tree/main/LICENSE"
    click Readme "https://github.com/sirioberati/webtwin/blob/main/README.md"

    %% Styles
    classDef ui fill:#3498DB,color:#fff,stroke:#2980B9,stroke-width:1px
    classDef server fill:#2ECC71,color:#fff,stroke:#27AE60,stroke-width:1px
    classDef core fill:#E67E22,color:#fff,stroke:#D35400,stroke-width:1px
    classDef external fill:#95A5A6,color:#fff,stroke:#7F8C8D,stroke-width:1px
```
