How sirioberati/webtwin works
Project Type and Purpose
• This is a Python-based full-stack web application (Flask) whose sole purpose is to provide an HTTP interface for extracting and packaging entire websites into a downloadable ZIP archive.
• The core “business logic” is an asset-extraction pipeline that can run in two modes: simple HTTP fetch (requests) or advanced JavaScript rendering (Selenium + Chrome WebDriver).
File Structure Analysis
Top-level files:
• app.py – Flask application entrypoint and HTTP routes
• setup.py / requirements.txt – packaging and dependencies
• app_architecture.md + docs/app_architecture_overview.md – textual and image descriptions of architecture
• templates/ – Jinja2 HTML templates for the UI (index.html: input form; components.html: detailed UI)
• LICENSE, CONTRIBUTING.md – repo governance
No database; no microservices. The architecture is a monolithic Flask service layered with template rendering and a modular extraction core.
README and Architecture Insights
• The README already spells out a multi-stage pipeline: URL submission → (optional) JS rendering → HTML parsing → asset discovery → parallel downloading → ZIP packaging → file download response.
• Key technologies: Flask (web server), requests (HTTP client), Selenium WebDriver (JS rendering), BeautifulSoup (HTML parsing), Python’s zipfile (ZIP creator).
Essential Details to Capture
• The bifurcation in the pipeline for advanced rendering (Selenium) vs. simple HTTP fetch.
• The parallel/concurrent nature of asset downloading.
• The packaging step and final HTTP response back to the client.
• The fact that there is no permanent storage–everything is generated on the fly and streamed.