# gyaanendra/webtwin architecture

> Architecture diagram of the GitHub repository gyaanendra/webtwin, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-23). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/gyaanendra/webtwin
- Repository: https://github.com/gyaanendra/webtwin

## Overview

##### Project Type and Purpose
- This is a full‑stack Python web application (“Website Extractor”) built with Flask.
- Purpose: Let users enter a URL via a browser UI, fetch and render the site (optionally with Selenium), parse HTML/CSS/JS/assets, download everything, package it into a ZIP, and return it for download.

##### File Structure Analysis
- Top level:
- app.py – Flask application entry point
- setup.py, requirements.txt – packaging and dependencies
- docs/ – Markdown docs including app_architecture_overview
- templates/ – HTML templates (index.html, components.html)
- app_architecture.md – detailed architecture write‑up
- No database; output is stored on the file system (ZIP files).
- Configuration handled via Flask and Selenium WebDriver.

##### Key Components
a. User & Browser UI (Frontend)
- Browser UI (templates/index.html)
- HTTP requests to Flask server
b. Flask Web Server (Presentation & Controller)
- Routes in app.py
- Handles form submission (“Extract Website”)
c. Extraction Core (Business Logic) – a modular subsystem with four subcomponents:
1. HTTP Client (requests library)
2. Selenium Renderer (Chrome WebDriver) – optional JS rendering
3. Content Parser (BeautifulSoup)
4. Asset Saver/Downloader + ZIP Creator
d. File System / Storage
- Temporary storage for downloaded assets
- Generated ZIP archives
e. External Dependencies
- Target website(s) on the internet
- Chrome/Chromium browser via Selenium WebDriver

##### Relationships & Data Flows
1. Browser → Flask server (URL & options)
2. Flask server → Extraction Core
3. Extraction Core → (a) HTTP Client or (b) Selenium Renderer → returns HTML
4. HTML → Content Parser → asset URLs
5. Asset URLs → Asset Downloader → local file storage
6. All assets → ZIP Creator → final archive
7. Flask server → Browser (serves the ZIP download)

##### Architectural Patterns & Design Principles
- Modular layering: Presentation (Flask) vs Business Logic (Extraction Core) vs Storage
- Single‑responsibility for each core component
- Optional pipeline branch for dynamic JS rendering
- Parallel/concurrent downloading (can annotate as “multi‑threaded” or “async”)

##### Technologies & Frameworks
- Flask (web framework)
- requests (HTTP client)
- Selenium WebDriver + ChromeDriver (JS rendering)
- BeautifulSoup (HTML parsing)
- Python’s zipfile module (archiving)
- Jinja2 templates (UI)

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Presentation Layer
    subgraph "Presentation Layer"
        direction TB
        BrowserUI["Browser UI\n(templates/index.html)"]:::frontend
        ComponentsPage["Components Page\n(templates/components.html)"]:::frontend
        FlaskServer["Flask Web Server\n(app.py)"]:::frontend
        RouteExtract["Route: /extract"]:::frontend
    end

    %% Extraction Core
    subgraph "Extraction Core" 
        direction TB
        HTTPClient["HTTP Client\n(requests)"]:::core
        Decision{"Advanced rendering?"}:::decision
        SeleniumRenderer["Selenium Renderer\n(ChromeDriver)"]:::core
        ContentParser["Content Parser\n(BeautifulSoup)"]:::core
        AssetSaver["Asset Saver &\nZIP Creator"]:::core
    end

    %% Storage Layer
    subgraph "Storage (File System)" 
        direction TB
        FileSystem["Temporary Folders &\nZIP Output"]:::storage
    end

    %% External Dependencies
    subgraph "External Dependencies"
        direction TB
        TargetSite["Target Website(s)"]:::external
        ChromeDriverExt["Chrome/Chromium\nvia Selenium"]:::external
    end

    %% Documentation & Config
    subgraph "Docs & Config"
        direction TB
        Config["setup.py"]:::external
        Dependencies["requirements.txt"]:::external
        ArchDoc["app_architecture.md"]:::external
        ArchOverview["docs/app_architecture_overview.md"]:::external
        README["README.md"]:::external
        CONTRIBUTING["CONTRIBUTING.md"]:::external
        LICENSE["LICENSE"]:::external
    end

    %% Flows
    BrowserUI -->|"URL + options"| FlaskServer
    ComponentsPage --> FlaskServer
    FlaskServer -->|"/extract"| RouteExtract
    RouteExtract -->|"calls core"| HTTPClient
    HTTPClient --> Decision
    Decision -->|No| ContentParser
    Decision -->|Yes| SeleniumRenderer
    SeleniumRenderer --> ContentParser
    ContentParser -->|"Asset URLs"| AssetSaver
    AssetSaver -->|"Files"| FileSystem
    FileSystem --> FlaskServer
    FlaskServer -->|"ZIP file"| BrowserUI
    TargetSite --> HTTPClient
    ChromeDriverExt --> SeleniumRenderer

    %% Legend
    subgraph "Legend"
        direction TB
        A[UI Layer]:::frontend
        B[Core Logic]:::core
        C[External]:::external
        D[Storage]:::storage
    end

    %% Click Events
    click BrowserUI "https://github.com/gyaanendra/webtwin/blob/main/templates/index.html"
    click ComponentsPage "https://github.com/gyaanendra/webtwin/blob/main/templates/components.html"
    click FlaskServer "https://github.com/gyaanendra/webtwin/blob/main/app.py"
    click RouteExtract "https://github.com/gyaanendra/webtwin/blob/main/app.py"
    click HTTPClient "https://github.com/gyaanendra/webtwin/blob/main/requirements.txt"
    click SeleniumRenderer "https://github.com/gyaanendra/webtwin/blob/main/requirements.txt"
    click ContentParser "https://github.com/gyaanendra/webtwin/blob/main/requirements.txt"
    click AssetSaver "https://github.com/gyaanendra/webtwin/blob/main/app.py"
    click Config "https://github.com/gyaanendra/webtwin/blob/main/setup.py"
    click Dependencies "https://github.com/gyaanendra/webtwin/blob/main/requirements.txt"
    click ArchDoc "https://github.com/gyaanendra/webtwin/blob/main/app_architecture.md"
    click ArchOverview "https://github.com/gyaanendra/webtwin/blob/main/docs/app_architecture_overview.md"
    click README "https://github.com/gyaanendra/webtwin/blob/main/README.md"
    click CONTRIBUTING "https://github.com/gyaanendra/webtwin/blob/main/CONTRIBUTING.md"
    click LICENSE "https://github.com/gyaanendra/webtwin/blob/main/LICENSE.md"

    %% Styles
    classDef frontend fill:#ADD8E6,stroke:#333,stroke-width:1px
    classDef core fill:#90EE90,stroke:#333,stroke-width:1px
    classDef external fill:#FFA500,stroke:#333,stroke-width:1px
    classDef storage fill:#D3D3D3,stroke:#333,stroke-width:1px
    classDef decision fill:#EEEEEE,stroke:#333,stroke-width:1px,stroke-dasharray: 5 5
```
