How ab-anand/scrapping works
This repository is a collection of independent Python, JavaScript, and Symfony web-scraping programs, not one unified application. The documented workflow is to retrieve public web content, extract useful records, and—in selected scrapers—save or export the results. The BoardGameGeek subproject adds a fuller import workflow with cached API data and locally stored XML and images; Meetup scraping writes parsed event dates to MongoDB. Several standalone scrapers are indexed but their source was not excerpted, so their internal steps and connections are not expanded.
Main entry points: GetBGGDataHandler.php BGG data handler · main.js Meetup RSS parser · SyncBGGData.php BGG sync command · ResizeImages.php Resize command
Components 34
Web scrapers
- EPL teams
epl_team.py - Hacker News
hacker_news.py - Snapdeal Python
snapdeal_spider.py - Snapdeal Cheerio
snapdeal_spider.js - Flipkart products
scrape_flipkart.py - Stack Overflow
webcrawler_stack.py - G2 reviews
webcrawler_g2.py - Cinema showtimes
21cinema.py - Expedia
expedia_scrap.py - WhatsApp Web
Whatsapp Web Scrapping/scraper.py - Meetup RSS parser
meetup_scrap/main.js
Content scrapers
- Book scraper
bookScrap.js - IMDb Top 250
imdb_top250/imdb.py - MyAnimeList
scrape_myanimelist/myanimelist.py - Rendered pages
javascript rendered web pages/day_1.py - Dropdown pages
javascript rendered web pages/dropdown_scrapping.py - Remote camera
scrapeCamera/read_remote_camera.py - SOK scraper
sok.js - Dataset scraper
dataset/ss.py
BoardGameGeek import
- BGG sync command
boardgamegeek-scraper/src/Command/Console/SyncBGGData.php - BGG request
boardgamegeek-scraper/src/Command/GetBGGData.php - BGG data handler
boardgamegeek-scraper/src/Command/GetBGGDataHandler.php - Rate-limit error
boardgamegeek-scraper/src/Exception/BoardGameGeekUnavailableException.php - Board game record
boardgamegeek-scraper/src/ValueObject/BoardGame.php - Resize command
boardgamegeek-scraper/src/Command/Console/ResizeImages.php - Resize options
boardgamegeek-scraper/src/Command/ResizeImage.php - Image resize handler
boardgamegeek-scraper/src/Command/ResizeImageHandler.php
Data outputs
- MongoDB
- Meetup group list
meetup_scrap/usergroups.json - Raw XML cache
- XML and images
boardgamegeek-scraper/public/boardgames
Other components
- Scraper user
- Meetup RSS feeds
- BoardGameGeek API
Connections 23
- Scraper user → EPL teams: runs scraper
README.md - Scraper user → Hacker News: runs scraper
README.md - Scraper user → Snapdeal Python: runs scraper
README.md - Scraper user → Snapdeal Cheerio: runs scraper
README.md - Scraper user → Flipkart products: runs scraper
README.md - Scraper user → Stack Overflow: searches term
README.md - Scraper user → G2 reviews: runs crawler
README.md - Scraper user → Cinema showtimes: gets showtimes
README.md - Meetup group list → Meetup RSS parser: provides groups
meetup_scrap/main.js - Meetup RSS parser → Meetup RSS feeds: fetches feeds
meetup_scrap/main.js - Meetup RSS parser → MongoDB: inserts dates
meetup_scrap/main.js - Scraper user → BGG sync command: starts sync
boardgamegeek-scraper/src/Command/Console/SyncBGGData.php - BGG sync command → BGG request: creates request
boardgamegeek-scraper/src/Command/Console/SyncBGGData.php - BGG sync command → BGG data handler: calls handler
boardgamegeek-scraper/src/Command/Console/SyncBGGData.php - BGG data handler → BoardGameGeek API: requests game data
boardgamegeek-scraper/src/Command/GetBGGDataHandler.php - BGG data handler → Raw XML cache: reads and writes
boardgamegeek-scraper/src/Command/GetBGGDataHandler.php - BGG data handler → Rate-limit error: throws on limit
boardgamegeek-scraper/src/Command/GetBGGDataHandler.php - BGG data handler → Board game record: creates records
boardgamegeek-scraper/src/Command/GetBGGDataHandler.php - BGG data handler → XML and images: saves XML and images
boardgamegeek-scraper/src/Command/GetBGGDataHandler.php - Scraper user → Resize command: starts resize
boardgamegeek-scraper/src/Command/Console/ResizeImages.php - Resize command → Resize options: creates options
boardgamegeek-scraper/src/Command/Console/ResizeImages.php - Resize command → Image resize handler: calls handler
boardgamegeek-scraper/src/Command/Console/ResizeImages.php - Image resize handler → XML and images: reads and writes images
boardgamegeek-scraper/src/Command/ResizeImageHandler.php