# syedhissamkazmi/web-scraper-studio architecture

> Architecture diagram of the GitHub repository syedhissamkazmi/web-scraper-studio, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/syedhissamkazmi/web-scraper-studio
- Repository: https://github.com/syedhissamkazmi/web-scraper-studio

## Overview

Web Scraper Studio is a Streamlit web app for extracting structured page data and exporting it. The main flow accepts one or more URLs, invokes the scraper, builds and optionally cleans a table, then presents results and downloads. The README documents additional views and capabilities; the sampled UI excerpt does not show every implementation detail, so uncertain internal wiring is omitted. Session history is held in Streamlit state, not a persistent database.

Main entry points: [`streamlit_app.py`](https://github.com/syedhissamkazmi/web-scraper-studio/blob/main/streamlit_app.py) (Web app), [`app/scraper.py`](https://github.com/syedhissamkazmi/web-scraper-studio/blob/main/app/scraper.py) (Scrape entry), [`app/analyzer.py`](https://github.com/syedhissamkazmi/web-scraper-studio/blob/main/app/analyzer.py) (Table building)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_interface["Streamlit interface"]
  node_ui["Web app<br/>[streamlit_app.py]"]
  node_batch["Batch input<br/>[streamlit_app.py]"]
  node_history["Session history<br/>[streamlit_app.py]"]
  node_results["Results views<br/>[streamlit_app.py]"]
  node_downloads["Downloads<br/>[streamlit_app.py]"]
end

subgraph group_scraping["Page scraping"]
  node_scrape["Scrape entry<br/>[scraper.py]"]
  node_normalize["URL normalization<br/>[scraper.py]"]
  node_robots["Robots check<br/>[scraper.py]"]
  node_fetch["HTTP fetch<br/>[scraper.py]"]
  node_parse["HTML extraction<br/>[scraper.py]"]
end

subgraph group_analysis["Analysis and export"]
  node_table["Table building<br/>[analyzer.py]"]
  node_clean["Data cleaning<br/>[analyzer.py]"]
  node_keywords["Keyword analysis<br/>[analyzer.py]"]
  node_contacts["Contact extraction<br/>[analyzer.py]"]
  node_excel["Excel encoding<br/>[analyzer.py]"]
end

node_user(("User"))
node_web(("Target website"))
node_robotsfile(("robots.txt"))

node_user -->|"enters URLs"| node_ui
node_ui -->|"selects mode"| node_batch
node_ui -->|"scrapes URLs"| node_scrape
node_scrape -->|"normalizes URL"| node_normalize
node_scrape -->|"checks optionally"| node_robots
node_robots -->|"reads policy"| node_robotsfile
node_scrape -->|"fetches page"| node_fetch
node_fetch -->|"requests HTML"| node_web
node_web -->|"returns response"| node_fetch
node_scrape -->|"parses content"| node_parse
node_ui -->|"builds table"| node_table
node_ui -->|"cleans optionally"| node_clean
node_ui -->|"records scans"| node_history
node_ui -->|"renders results"| node_results
node_results -->|"analyzes content"| node_keywords
node_results -->|"extracts contacts"| node_contacts
node_ui -->|"offers formats"| node_downloads
node_downloads -->|"encodes workbook"| node_excel

click node_ui "https://github.com/syedhissamkazmi/web-scraper-studio/blob/main/streamlit_app.py"
click node_batch "https://github.com/syedhissamkazmi/web-scraper-studio/blob/main/streamlit_app.py"
click node_history "https://github.com/syedhissamkazmi/web-scraper-studio/blob/main/streamlit_app.py"
click node_results "https://github.com/syedhissamkazmi/web-scraper-studio/blob/main/streamlit_app.py"
click node_downloads "https://github.com/syedhissamkazmi/web-scraper-studio/blob/main/streamlit_app.py"
click node_scrape "https://github.com/syedhissamkazmi/web-scraper-studio/blob/main/app/scraper.py"
click node_normalize "https://github.com/syedhissamkazmi/web-scraper-studio/blob/main/app/scraper.py"
click node_robots "https://github.com/syedhissamkazmi/web-scraper-studio/blob/main/app/scraper.py"
click node_fetch "https://github.com/syedhissamkazmi/web-scraper-studio/blob/main/app/scraper.py"
click node_parse "https://github.com/syedhissamkazmi/web-scraper-studio/blob/main/app/scraper.py"
click node_table "https://github.com/syedhissamkazmi/web-scraper-studio/blob/main/app/analyzer.py"
click node_clean "https://github.com/syedhissamkazmi/web-scraper-studio/blob/main/app/analyzer.py"
click node_keywords "https://github.com/syedhissamkazmi/web-scraper-studio/blob/main/app/analyzer.py"
click node_contacts "https://github.com/syedhissamkazmi/web-scraper-studio/blob/main/app/analyzer.py"
click node_excel "https://github.com/syedhissamkazmi/web-scraper-studio/blob/main/app/analyzer.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_ui,node_batch,node_history,node_results,node_downloads,node_user toneBlue
class node_scrape,node_normalize,node_robots,node_fetch,node_parse toneAmber
class node_table,node_clean,node_keywords,node_contacts,node_excel toneMint
class node_web,node_robotsfile toneIndigo
```

## Components

### Streamlit interface

- [Web app](https://github.com/syedhissamkazmi/web-scraper-studio/blob/main/streamlit_app.py): `streamlit_app.py`
- [Batch input](https://github.com/syedhissamkazmi/web-scraper-studio/blob/main/streamlit_app.py): `streamlit_app.py`
- [Session history](https://github.com/syedhissamkazmi/web-scraper-studio/blob/main/streamlit_app.py): `streamlit_app.py`
- [Results views](https://github.com/syedhissamkazmi/web-scraper-studio/blob/main/streamlit_app.py): `streamlit_app.py`
- [Downloads](https://github.com/syedhissamkazmi/web-scraper-studio/blob/main/streamlit_app.py): `streamlit_app.py`

### Page scraping

- [Scrape entry](https://github.com/syedhissamkazmi/web-scraper-studio/blob/main/app/scraper.py): `app/scraper.py`
- [URL normalization](https://github.com/syedhissamkazmi/web-scraper-studio/blob/main/app/scraper.py): `app/scraper.py`
- [Robots check](https://github.com/syedhissamkazmi/web-scraper-studio/blob/main/app/scraper.py): `app/scraper.py`
- [HTTP fetch](https://github.com/syedhissamkazmi/web-scraper-studio/blob/main/app/scraper.py): `app/scraper.py`
- [HTML extraction](https://github.com/syedhissamkazmi/web-scraper-studio/blob/main/app/scraper.py): `app/scraper.py`

### Analysis and export

- [Table building](https://github.com/syedhissamkazmi/web-scraper-studio/blob/main/app/analyzer.py): `app/analyzer.py`
- [Data cleaning](https://github.com/syedhissamkazmi/web-scraper-studio/blob/main/app/analyzer.py): `app/analyzer.py`
- [Keyword analysis](https://github.com/syedhissamkazmi/web-scraper-studio/blob/main/app/analyzer.py): `app/analyzer.py`
- [Contact extraction](https://github.com/syedhissamkazmi/web-scraper-studio/blob/main/app/analyzer.py): `app/analyzer.py`
- [Excel encoding](https://github.com/syedhissamkazmi/web-scraper-studio/blob/main/app/analyzer.py): `app/analyzer.py`

### Other components

- **User**
- **Target website**
- **robots.txt**

## Connections

- User → Web app: enters URLs
- Web app → Batch input: selects mode
- Web app → Scrape entry: scrapes URLs
- Scrape entry → URL normalization: normalizes URL
- Scrape entry → Robots check: checks optionally
- Robots check → robots.txt: reads policy
- Scrape entry → HTTP fetch: fetches page
- HTTP fetch → Target website: requests HTML
- Target website → HTTP fetch: returns response
- Scrape entry → HTML extraction: parses content
- Web app → Table building: builds table
- Web app → Data cleaning: cleans optionally
- Web app → Session history: records scans
- Web app → Results views: renders results
- Results views → Keyword analysis: analyzes content
- Results views → Contact extraction: extracts contacts
- Web app → Downloads: offers formats
- Downloads → Excel encoding: encodes workbook