How syedhissamkazmi/web-scraper-studio works
Web Scraper Studio is a Streamlit web app for extracting structured page data and exporting it. The main flow accepts one or more URLs, invokes the scraper, builds and optionally cleans a table, then presents results and downloads. The README documents additional views and capabilities; the sampled UI excerpt does not show every implementation detail, so uncertain internal wiring is omitted. Session history is held in Streamlit state, not a persistent database.
Main entry points: streamlit_app.py Web app · scraper.py Scrape entry · analyzer.py Table building
Components 18
Streamlit interface
- Web app
streamlit_app.py - Batch input
streamlit_app.py - Session history
streamlit_app.py - Results views
streamlit_app.py - Downloads
streamlit_app.py
Page scraping
- Scrape entry
app/scraper.py - URL normalization
app/scraper.py - Robots check
app/scraper.py - HTTP fetch
app/scraper.py - HTML extraction
app/scraper.py
Analysis and export
- Table building
app/analyzer.py - Data cleaning
app/analyzer.py - Keyword analysis
app/analyzer.py - Contact extraction
app/analyzer.py - Excel encoding
app/analyzer.py
Other components
- User
- Target website
- robots.txt
Connections 18
- User → Web app: enters URLs
- Web app → Batch input: selects mode
- Web app → Scrape entry: scrapes URLs
- Scrape entry → URL normalization: normalizes URL
- Scrape entry → Robots check: checks optionally
- Robots check → robots.txt: reads policy
- Scrape entry → HTTP fetch: fetches page
- HTTP fetch → Target website: requests HTML
- Target website → HTTP fetch: returns response
- Scrape entry → HTML extraction: parses content
- Web app → Table building: builds table
- Web app → Data cleaning: cleans optionally
- Web app → Session history: records scans
- Web app → Results views: renders results
- Results views → Keyword analysis: analyzes content
- Results views → Contact extraction: extracts contacts
- Web app → Downloads: offers formats
- Downloads → Excel encoding: encodes workbook