How eslamaboushashaa/scrape-facebook-by-playwright works
Project type and purpose
• This is a lightweight full-stack Python application whose job is to scrape Facebook page metrics, analyze them, and present insights via a web interface.
File structure
• Top-level files:
• facebook_backend.py – contains the scraping and analysis logic
• web_app.py — implements the Streamlit UI that calls into the backend
• requirements.txt — lists Python dependencies (Playwright, BeautifulSoup, pandas, Streamlit, etc.)
• setup.sh — bootstraps the environment (installs system deps, Python packages)
• README.md — describes features and tech stack
README for additional insights
• Core features:
• Scrape post date, reactions, comments, shares from public and private pages via Playwright + BeautifulSoup
• Load data into pandas DataFrame, clean and analyze it
• Render graphs and tables in a Streamlit web app for competitor analysis
• Technical stack: Python, OOP, Playwright, BeautifulSoup, pandas, Streamlit
a. Main components
1) External Data Source: Facebook pages (public/private)
2) Scraper module (Playwright)
3) Parser module (BeautifulSoup)
4) Data processor (pandas analytics)
5) Web UI (Streamlit)
6) Environment/orchestration (setup.sh, Python runtime)
b. Relationships & data flow
• User opens the Streamlit UI (web_app.py)
→ UI triggers a “scrape and analyze” request
→ web_app.py calls functions in facebook_backend.py
→ Scraper module uses Playwright to fetch page HTML
→ Parser (BeautifulSoup) extracts metrics into raw data
→ Data processor (pandas) cleans, aggregates, and computes insights
→ Processed data and charts are returned to web_app.py
→ Streamlit renders tables and graphs for the user
c. Architectural patterns & design principles
• Layered architecture:
– Presentation layer (Streamlit front-end)
– Service layer (facebook_backend.py orchestrates scraping + analysis)
– Infrastructure layer (external Facebook pages, environment setup)
• Single-responsibility modules:
– Scraping vs. parsing vs. data analysis vs. UI rendering each live in their own functions or submodules
• Minimal external dependencies—everything is managed via requirements.txt
d. Key technologies & frameworks
• Playwright: headless browsing, logging in (if needed), HTML retrieval
• BeautifulSoup: HTML parsing and data extraction
• pandas: DataFrame creation, cleaning, analysis, pivoting
• Streamlit: rapidly build and serve interactive web dashboards
• setup.sh: reproducible setup of Python virtualenv and system prerequisites
Putting it all together
• At the top, show “User Web Browser” → Streamlit UI
• Right of service layer: chain of Scraper → Parser → Data Processor
• Bottom: “Hosting/Deployment” environment block (setup.sh, virtualenv, Python runtime)