How quentinadt/check-impact-of-blocking-ressources works
Project type and purpose
• From the file tree (main.py, README.md) and README, this is a small full‑stack Python web application (Flask) that drives headless browser sessions (Playwright) to simulate Googlebot Mobile, blocks resources per user‑defined rules, captures screenshots, and displays results in a simple web UI. Its purpose is Technical SEO impact analysis (pre‑launch QA, robots.txt validation, visual regression detection).
File structure
• Top level:
• main.py – single entrypoint: defines Flask routes, orchestrates test runs, invokes Playwright and robots.txt checks, writes screenshots, streams logs.
• README.md – describes features, setup, usage.
• .gitignore – excludes browser binaries, screenshots, etc.
• No separate “frontend” folder: the UI is embedded in Flask templates/static (likely defined in main.py).
• External config comes from CLI flags and form inputs (URL, blocking mode, predefined list).
README for additional insights
• Core libraries and services:
• Flask (web server + UI)
• Playwright (Chromium headless sessions)
• gpyrobotstxt (robots.txt parser for Googlebot)
• Concurrency: runs blocking tests in parallel batches.
• Storage: local filesystem directory (screenshots_playwright).
• Live logs streamed over WebSockets or server‐sent events (implied).
a. Main components:
1. Web Client (Browser) – user interacts with HTML form, views live logs and final screenshots.
2. Flask Application – handles HTTP requests, serves UI, accepts form data, exposes WebSocket/SSE for log streaming.
3. Test Orchestrator Module – processes inputs, builds blocking scenarios, enqueues jobs.
4. Playwright Worker Pool – headless Chromium instances that load pages under different resource‑blocking rules.
5. Robots.txt Checker – validates each blocked resource’s path against Googlebot rules.
6. Screenshot Manager – saves screenshots to screenshots_playwright, keeps metadata.
7. Logger – streams real‑time logs back to the Web Client.
8. File System – persists screenshots and logs.
9. (Optional) Task Queue – if concurrency is managed via Celery/RQ or Python threads/process pools.
b. Relationships and data flows:
• User ➔ HTTP POST ➔ Flask App “Start Tests” route
• Flask App ➔ Test Orchestrator (in‑process or via Task Queue)
• Test Orchestrator ➔ Playwright Worker Pool (parallel jobs), passing blocking rules + URL + Googlebot UA
• Each Worker ➔
– Load robots.txt via Robots.txt Checker
– Navigate page with blocked resources
– Capture screenshot via Playwright API
– Report status/results back to Orchestrator
• Orchestrator ➔ Screenshot Manager (save files + metadata)
• Orchestrator ➔ Logger (emit live logs/events) ➔ Web Client
• After all jobs, Flask App aggregates results, renders final HTML page.
c. Architectural patterns and principles:
• MVC‑style layering:
• Model = test definitions + screenshot metadata on disk
• View = Flask templates + live logs front end
• Controller = Flask routes + Orchestrator logic
• Concurrency pattern: worker pool or task queue for parallel Playwright sessions
• Separation of concerns: HTTP/UI layer vs. orchestration vs. browser automation vs. storage.
d. Key technologies & libraries:
• Flask (Python web framework)
• Playwright (headless Chromium automation)
• gpyrobotstxt (robots.txt parsing)
• Python’s concurrent.futures or a task queue (for parallelism)
• Filesystem storage (screenshots_playwright)