How gyaanendra/webtwin works
Project Type and Purpose
• This is a full‑stack Python web application (“Website Extractor”) built with Flask.
• Purpose: Let users enter a URL via a browser UI, fetch and render the site (optionally with Selenium), parse HTML/CSS/JS/assets, download everything, package it into a ZIP, and return it for download.
File Structure Analysis
• Top level:
• app.py – Flask application entry point
• setup.py, requirements.txt – packaging and dependencies
• docs/ – Markdown docs including app_architecture_overview
• templates/ – HTML templates (index.html, components.html)
• app_architecture.md – detailed architecture write‑up
• No database; output is stored on the file system (ZIP files).
• Configuration handled via Flask and Selenium WebDriver.
Key Components
a. User & Browser UI (Frontend)
• Browser UI (templates/index.html)
• HTTP requests to Flask server
b. Flask Web Server (Presentation & Controller)
• Routes in app.py
• Handles form submission (“Extract Website”)
c. Extraction Core (Business Logic) – a modular subsystem with four subcomponents:
1. HTTP Client (requests library)
2. Selenium Renderer (Chrome WebDriver) – optional JS rendering
3. Content Parser (BeautifulSoup)
4. Asset Saver/Downloader + ZIP Creator
d. File System / Storage
• Temporary storage for downloaded assets
• Generated ZIP archives
e. External Dependencies
• Target website(s) on the internet
• Chrome/Chromium browser via Selenium WebDriver
Relationships & Data Flows
1. Browser → Flask server (URL & options)
2. Flask server → Extraction Core
3. Extraction Core → (a) HTTP Client or (b) Selenium Renderer → returns HTML
4. HTML → Content Parser → asset URLs
5. Asset URLs → Asset Downloader → local file storage
6. All assets → ZIP Creator → final archive
7. Flask server → Browser (serves the ZIP download)
Architectural Patterns & Design Principles
• Modular layering: Presentation (Flask) vs Business Logic (Extraction Core) vs Storage
• Single‑responsibility for each core component
• Optional pipeline branch for dynamic JS rendering
• Parallel/concurrent downloading (can annotate as “multi‑threaded” or “async”)
Technologies & Frameworks
• Flask (web framework)
• requests (HTTP client)
• Selenium WebDriver + ChromeDriver (JS rendering)
• BeautifulSoup (HTML parsing)
• Python’s zipfile module (archiving)
• Jinja2 templates (UI)