# ab-anand/scrapping architecture

> Architecture diagram of the GitHub repository ab-anand/scrapping, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-30). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/ab-anand/scrapping
- Repository: https://github.com/ab-anand/scrapping

## Overview

This repository is a collection of independent Python, JavaScript, and Symfony web-scraping programs, not one unified application. The documented workflow is to retrieve public web content, extract useful records, and—in selected scrapers—save or export the results. The BoardGameGeek subproject adds a fuller import workflow with cached API data and locally stored XML and images; Meetup scraping writes parsed event dates to MongoDB. Several standalone scrapers are indexed but their source was not excerpted, so their internal steps and connections are not expanded.

Main entry points: [`boardgamegeek-scraper/src/Command/GetBGGDataHandler.php`](https://github.com/ab-anand/scrapping/blob/master/boardgamegeek-scraper/src/Command/GetBGGDataHandler.php) (BGG data handler), [`meetup_scrap/main.js`](https://github.com/ab-anand/scrapping/blob/master/meetup_scrap/main.js) (Meetup RSS parser), [`boardgamegeek-scraper/src/Command/Console/SyncBGGData.php`](https://github.com/ab-anand/scrapping/blob/master/boardgamegeek-scraper/src/Command/Console/SyncBGGData.php) (BGG sync command), [`boardgamegeek-scraper/src/Command/Console/ResizeImages.php`](https://github.com/ab-anand/scrapping/blob/master/boardgamegeek-scraper/src/Command/Console/ResizeImages.php) (Resize command)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_web_scrapers["Web scrapers"]
  node_epl["EPL teams<br/>[epl_team.py]"]
  node_hacker["Hacker News<br/>[hacker_news.py]"]
  node_snapdeal_py["Snapdeal Python<br/>[snapdeal_spider.py]"]
  node_snapdeal_js["Snapdeal Cheerio<br/>[snapdeal_spider.js]"]
  node_flipkart["Flipkart products<br/>[scrape_flipkart.py]"]
  node_stack["Stack Overflow"]
  node_g2["G2 reviews<br/>[webcrawler_g2.py]"]
  node_cinema["Cinema showtimes<br/>[21cinema.py]"]
  node_expedia["Expedia<br/>[expedia_scrap.py]"]
  node_whatsapp["WhatsApp Web<br/>[scraper.py]"]
  node_meetup["Meetup RSS parser<br/>[main.js]"]
end

subgraph group_content_scrapers["Content scrapers"]
  node_book["Book scraper<br/>[bookScrap.js]"]
  node_imdb["IMDb Top 250<br/>[imdb.py]"]
  node_anime["MyAnimeList<br/>[myanimelist.py]"]
  node_rendered["Rendered pages<br/>[day_1.py]"]
  node_dropdown["Dropdown pages"]
  node_camera["Remote camera"]
  node_sok["SOK scraper<br/>[sok.js]"]
  node_dataset["Dataset scraper<br/>[ss.py]"]
end

subgraph group_bgg_import["BoardGameGeek import"]
  node_bgg_cli["BGG sync command<br/>[SyncBGGData.php]"]
  node_bgg_request["BGG request<br/>[GetBGGData.php]"]
  node_bgg_handler["BGG data handler"]
  node_bgg_exception["Rate-limit error"]
  node_boardgame["Board game record<br/>[BoardGame.php]"]
  node_resize_cli["Resize command<br/>[ResizeImages.php]"]
  node_resize_request["Resize options<br/>[ResizeImage.php]"]
  node_resize_handler["Image resize handler"]
end

subgraph group_data_outputs["Data outputs"]
  node_mongo[("MongoDB")]
  node_meetup_json["Meetup group list<br/>[usergroups.json]"]
  node_bgg_cache[("Raw XML cache")]
  node_bgg_files["XML and images"]
end

node_user(("Scraper user"))
node_meetup_source["Meetup RSS feeds"]
node_bgg_service["BoardGameGeek API"]

node_user -->|"runs scraper"| node_epl
node_user -->|"runs scraper"| node_hacker
node_user -->|"runs scraper"| node_snapdeal_py
node_user -->|"runs scraper"| node_snapdeal_js
node_user -->|"runs scraper"| node_flipkart
node_user -->|"searches term"| node_stack
node_user -->|"runs crawler"| node_g2
node_user -->|"gets showtimes"| node_cinema
node_meetup_json -->|"provides groups"| node_meetup
node_meetup -->|"fetches feeds"| node_meetup_source
node_meetup -->|"inserts dates"| node_mongo
node_user -->|"starts sync"| node_bgg_cli
node_bgg_cli -->|"creates request"| node_bgg_request
node_bgg_cli -->|"calls handler"| node_bgg_handler
node_bgg_handler -->|"requests game data"| node_bgg_service
node_bgg_handler -->|"reads and writes"| node_bgg_cache
node_bgg_handler -->|"throws on limit"| node_bgg_exception
node_bgg_handler -->|"creates records"| node_boardgame
node_bgg_handler -->|"saves XML and images"| node_bgg_files
node_user -->|"starts resize"| node_resize_cli
node_resize_cli -->|"creates options"| node_resize_request
node_resize_cli -->|"calls handler"| node_resize_handler
node_resize_handler -->|"reads and writes images"| node_bgg_files

click node_epl "https://github.com/ab-anand/scrapping/blob/master/epl_team.py"
click node_hacker "https://github.com/ab-anand/scrapping/blob/master/hacker_news.py"
click node_snapdeal_py "https://github.com/ab-anand/scrapping/blob/master/snapdeal_spider.py"
click node_snapdeal_js "https://github.com/ab-anand/scrapping/blob/master/snapdeal_spider.js"
click node_flipkart "https://github.com/ab-anand/scrapping/blob/master/scrape_flipkart.py"
click node_stack "https://github.com/ab-anand/scrapping/blob/master/webcrawler_stack.py"
click node_g2 "https://github.com/ab-anand/scrapping/blob/master/webcrawler_g2.py"
click node_cinema "https://github.com/ab-anand/scrapping/blob/master/21cinema.py"
click node_expedia "https://github.com/ab-anand/scrapping/blob/master/expedia_scrap.py"
click node_whatsapp "https://github.com/ab-anand/scrapping/blob/master/Whatsapp%20Web%20Scrapping/scraper.py"
click node_meetup "https://github.com/ab-anand/scrapping/blob/master/meetup_scrap/main.js"
click node_meetup_json "https://github.com/ab-anand/scrapping/blob/master/meetup_scrap/usergroups.json"
click node_book "https://github.com/ab-anand/scrapping/blob/master/bookScrap.js"
click node_imdb "https://github.com/ab-anand/scrapping/blob/master/imdb_top250/imdb.py"
click node_anime "https://github.com/ab-anand/scrapping/blob/master/scrape_myanimelist/myanimelist.py"
click node_rendered "https://github.com/ab-anand/scrapping/blob/master/javascript%20rendered%20web%20pages/day_1.py"
click node_dropdown "https://github.com/ab-anand/scrapping/blob/master/javascript%20rendered%20web%20pages/dropdown_scrapping.py"
click node_camera "https://github.com/ab-anand/scrapping/blob/master/scrapeCamera/read_remote_camera.py"
click node_sok "https://github.com/ab-anand/scrapping/blob/master/sok.js"
click node_dataset "https://github.com/ab-anand/scrapping/blob/master/dataset/ss.py"
click node_bgg_cli "https://github.com/ab-anand/scrapping/blob/master/boardgamegeek-scraper/src/Command/Console/SyncBGGData.php"
click node_bgg_request "https://github.com/ab-anand/scrapping/blob/master/boardgamegeek-scraper/src/Command/GetBGGData.php"
click node_bgg_handler "https://github.com/ab-anand/scrapping/blob/master/boardgamegeek-scraper/src/Command/GetBGGDataHandler.php"
click node_bgg_exception "https://github.com/ab-anand/scrapping/blob/master/boardgamegeek-scraper/src/Exception/BoardGameGeekUnavailableException.php"
click node_boardgame "https://github.com/ab-anand/scrapping/blob/master/boardgamegeek-scraper/src/ValueObject/BoardGame.php"
click node_bgg_files "https://github.com/ab-anand/scrapping/tree/master/boardgamegeek-scraper/public/boardgames"
click node_resize_cli "https://github.com/ab-anand/scrapping/blob/master/boardgamegeek-scraper/src/Command/Console/ResizeImages.php"
click node_resize_request "https://github.com/ab-anand/scrapping/blob/master/boardgamegeek-scraper/src/Command/ResizeImage.php"
click node_resize_handler "https://github.com/ab-anand/scrapping/blob/master/boardgamegeek-scraper/src/Command/ResizeImageHandler.php"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_epl,node_hacker,node_snapdeal_py,node_snapdeal_js,node_flipkart,node_stack,node_g2,node_cinema,node_expedia,node_whatsapp,node_meetup,node_user toneBlue
class node_book,node_imdb,node_anime,node_rendered,node_dropdown,node_camera,node_sok,node_dataset toneAmber
class node_bgg_cli,node_bgg_request,node_bgg_handler,node_bgg_exception,node_boardgame,node_resize_cli,node_resize_request,node_resize_handler,node_bgg_service toneMint
class node_mongo,node_meetup_json,node_bgg_cache,node_bgg_files toneRose
class node_meetup_source toneIndigo
```

## Components

### Web scrapers

- [EPL teams](https://github.com/ab-anand/scrapping/blob/master/epl_team.py): `epl_team.py`
- [Hacker News](https://github.com/ab-anand/scrapping/blob/master/hacker_news.py): `hacker_news.py`
- [Snapdeal Python](https://github.com/ab-anand/scrapping/blob/master/snapdeal_spider.py): `snapdeal_spider.py`
- [Snapdeal Cheerio](https://github.com/ab-anand/scrapping/blob/master/snapdeal_spider.js): `snapdeal_spider.js`
- [Flipkart products](https://github.com/ab-anand/scrapping/blob/master/scrape_flipkart.py): `scrape_flipkart.py`
- [Stack Overflow](https://github.com/ab-anand/scrapping/blob/master/webcrawler_stack.py): `webcrawler_stack.py`
- [G2 reviews](https://github.com/ab-anand/scrapping/blob/master/webcrawler_g2.py): `webcrawler_g2.py`
- [Cinema showtimes](https://github.com/ab-anand/scrapping/blob/master/21cinema.py): `21cinema.py`
- [Expedia](https://github.com/ab-anand/scrapping/blob/master/expedia_scrap.py): `expedia_scrap.py`
- [WhatsApp Web](https://github.com/ab-anand/scrapping/blob/master/Whatsapp%20Web%20Scrapping/scraper.py): `Whatsapp Web Scrapping/scraper.py`
- [Meetup RSS parser](https://github.com/ab-anand/scrapping/blob/master/meetup_scrap/main.js): `meetup_scrap/main.js`

### Content scrapers

- [Book scraper](https://github.com/ab-anand/scrapping/blob/master/bookScrap.js): `bookScrap.js`
- [IMDb Top 250](https://github.com/ab-anand/scrapping/blob/master/imdb_top250/imdb.py): `imdb_top250/imdb.py`
- [MyAnimeList](https://github.com/ab-anand/scrapping/blob/master/scrape_myanimelist/myanimelist.py): `scrape_myanimelist/myanimelist.py`
- [Rendered pages](https://github.com/ab-anand/scrapping/blob/master/javascript%20rendered%20web%20pages/day_1.py): `javascript rendered web pages/day_1.py`
- [Dropdown pages](https://github.com/ab-anand/scrapping/blob/master/javascript%20rendered%20web%20pages/dropdown_scrapping.py): `javascript rendered web pages/dropdown_scrapping.py`
- [Remote camera](https://github.com/ab-anand/scrapping/blob/master/scrapeCamera/read_remote_camera.py): `scrapeCamera/read_remote_camera.py`
- [SOK scraper](https://github.com/ab-anand/scrapping/blob/master/sok.js): `sok.js`
- [Dataset scraper](https://github.com/ab-anand/scrapping/blob/master/dataset/ss.py): `dataset/ss.py`

### BoardGameGeek import

- [BGG sync command](https://github.com/ab-anand/scrapping/blob/master/boardgamegeek-scraper/src/Command/Console/SyncBGGData.php): `boardgamegeek-scraper/src/Command/Console/SyncBGGData.php`
- [BGG request](https://github.com/ab-anand/scrapping/blob/master/boardgamegeek-scraper/src/Command/GetBGGData.php): `boardgamegeek-scraper/src/Command/GetBGGData.php`
- [BGG data handler](https://github.com/ab-anand/scrapping/blob/master/boardgamegeek-scraper/src/Command/GetBGGDataHandler.php): `boardgamegeek-scraper/src/Command/GetBGGDataHandler.php`
- [Rate-limit error](https://github.com/ab-anand/scrapping/blob/master/boardgamegeek-scraper/src/Exception/BoardGameGeekUnavailableException.php): `boardgamegeek-scraper/src/Exception/BoardGameGeekUnavailableException.php`
- [Board game record](https://github.com/ab-anand/scrapping/blob/master/boardgamegeek-scraper/src/ValueObject/BoardGame.php): `boardgamegeek-scraper/src/ValueObject/BoardGame.php`
- [Resize command](https://github.com/ab-anand/scrapping/blob/master/boardgamegeek-scraper/src/Command/Console/ResizeImages.php): `boardgamegeek-scraper/src/Command/Console/ResizeImages.php`
- [Resize options](https://github.com/ab-anand/scrapping/blob/master/boardgamegeek-scraper/src/Command/ResizeImage.php): `boardgamegeek-scraper/src/Command/ResizeImage.php`
- [Image resize handler](https://github.com/ab-anand/scrapping/blob/master/boardgamegeek-scraper/src/Command/ResizeImageHandler.php): `boardgamegeek-scraper/src/Command/ResizeImageHandler.php`

### Data outputs

- **MongoDB**
- [Meetup group list](https://github.com/ab-anand/scrapping/blob/master/meetup_scrap/usergroups.json): `meetup_scrap/usergroups.json`
- **Raw XML cache**
- [XML and images](https://github.com/ab-anand/scrapping/tree/master/boardgamegeek-scraper/public/boardgames): `boardgamegeek-scraper/public/boardgames`

### Other components

- **Scraper user**
- **Meetup RSS feeds**
- **BoardGameGeek API**

## Connections

- Scraper user → EPL teams: runs scraper (evidence: `README.md`)
- Scraper user → Hacker News: runs scraper (evidence: `README.md`)
- Scraper user → Snapdeal Python: runs scraper (evidence: `README.md`)
- Scraper user → Snapdeal Cheerio: runs scraper (evidence: `README.md`)
- Scraper user → Flipkart products: runs scraper (evidence: `README.md`)
- Scraper user → Stack Overflow: searches term (evidence: `README.md`)
- Scraper user → G2 reviews: runs crawler (evidence: `README.md`)
- Scraper user → Cinema showtimes: gets showtimes (evidence: `README.md`)
- Meetup group list → Meetup RSS parser: provides groups (evidence: `meetup_scrap/main.js`)
- Meetup RSS parser → Meetup RSS feeds: fetches feeds (evidence: `meetup_scrap/main.js`)
- Meetup RSS parser → MongoDB: inserts dates (evidence: `meetup_scrap/main.js`)
- Scraper user → BGG sync command: starts sync (evidence: `boardgamegeek-scraper/src/Command/Console/SyncBGGData.php`)
- BGG sync command → BGG request: creates request (evidence: `boardgamegeek-scraper/src/Command/Console/SyncBGGData.php`)
- BGG sync command → BGG data handler: calls handler (evidence: `boardgamegeek-scraper/src/Command/Console/SyncBGGData.php`)
- BGG data handler → BoardGameGeek API: requests game data (evidence: `boardgamegeek-scraper/src/Command/GetBGGDataHandler.php`)
- BGG data handler → Raw XML cache: reads and writes (evidence: `boardgamegeek-scraper/src/Command/GetBGGDataHandler.php`)
- BGG data handler → Rate-limit error: throws on limit (evidence: `boardgamegeek-scraper/src/Command/GetBGGDataHandler.php`)
- BGG data handler → Board game record: creates records (evidence: `boardgamegeek-scraper/src/Command/GetBGGDataHandler.php`)
- BGG data handler → XML and images: saves XML and images (evidence: `boardgamegeek-scraper/src/Command/GetBGGDataHandler.php`)
- Scraper user → Resize command: starts resize (evidence: `boardgamegeek-scraper/src/Command/Console/ResizeImages.php`)
- Resize command → Resize options: creates options (evidence: `boardgamegeek-scraper/src/Command/Console/ResizeImages.php`)
- Resize command → Image resize handler: calls handler (evidence: `boardgamegeek-scraper/src/Command/Console/ResizeImages.php`)
- Image resize handler → XML and images: reads and writes images (evidence: `boardgamegeek-scraper/src/Command/ResizeImageHandler.php`)
