How anderson93/oncase-scrapper-challenge works
Project type and purpose
• This is a command-line web crawling/scraping application built with Scrapy + BeautifulSoup in Python, targeting three tech-news portals (OlharDigital, CanalTech, GizModo).
• It orchestrates link-discovery spiders, detail-scraping spiders, processes the items through pipelines, and persists the results in a MongoDB database (tech_news).
File structure
• Top level:
• start.sh – entry-point shell script that prompts for “cold start” (invokes drop_tech_db.py) and then kicks off the Scrapy project.
• drop_tech_db.py – standalone script to drop the tech_news database.
• requirements.txt – Python dependencies.
• techcrawlers/ – contains scrapy.cfg and the actual Scrapy project.
• spiders/ – six spider classes (three *Links spiders* to discover article URLs, three *detail spiders* to scrape article content).
• items.py – defines the item schema.
• middlewares.py – custom downloader/request middlewares.
• pipelines.py – routes scraped items to backends.
• mongo_connector.py – helper for connecting to MongoDB.
• settings.py – Scrapy settings (concurrency, pipelines, middlewares, feed exports, logging).
README for additional insights
• Confirms use of MongoDB with four collections: gizmodo-links, olhardigital-links, canaltech-links, and news_data.
• Describes “cold start” logic, per-site page count input, and shows job output.
• Mentions optional scaling via Scrapyd/ScrapydWeb, monitoring, request throttling, header/cookie management.