How rugantio/fbcrawl works
Project type and purpose
• This is a Python web crawler for Facebook pages and posts, built on the Scrapy framework.
• Two main spiders: one for posts (“fb”) and one for comments.
File structure
• Top level:
– scrapy.cfg (Scrapy project configuration)
– runner_facebook.sh (example shell wrapper)
• fbcrawl package:
– settings.py (framework settings, pipelines, concurrency, robots.txt)
– middlewares.py (custom request/response hooks, login checkpoint handling)
– items.py (declaration of scraped data fields)
– pipelines.py (date-filter pipeline, CSV/JSON exporters)
– spiders/ (directory containing fbcrawl.py, comments.py, plus auxiliary spiders for events and profiles)
Extract architecture details from the README
• The Scrapy Engine orchestrates:
– Scheduler (queues Requests)
– Downloader & Downloader Middlewares (HTTPS requests to mbasic.facebook.com)
– Spider components (parse logic using lxml/XPath)
– Item Pipeline (post-processing, filtering, exporting)
• External dependency: Facebook’s mobile HTML interface (mbasic.facebook.com)
• Output: CSV or JSON via Scrapy’s Feed Exporter
Main components
a. Entrypoint
– CLI/runner_facebook.sh invoking “scrapy crawl fb|comments” with arguments.
b. Configuration
– scrapy.cfg + settings.py feeding into the Scrapy Engine.
– Scheduler
– Downloader (with custom Downloader Middlewares)
– Spiders (fb spider and comments spider)
– Item Pipeline (FbcrawlPipeline)
d. External service
– mbasic.facebook.com (Facebook mobile HTML interface)
e. Data storage
– CSV/JSON via Scrapy Feed Exporter
f. Supporting libraries
– Python3, Scrapy, Twisted, lxml, XPath
Relationships and data flow
1. User → CLI (runner_facebook.sh or scrapyd command)
2. CLI → Scrapy Engine (loads settings, instantiates scheduler, downloader, spiders, pipelines)
3. Scrapy Engine → Scheduler (enqueue initial Request)
4. Scheduler → Downloader → External Service (HTTP GET)
5. External Service → Downloader → Downloader Middlewares (login, cookies) → Spider.parse
6. Spider.parse → produces new Requests (cycle back to scheduler) and Items
7. Items → Item Pipeline (FbcrawlPipeline) → filtering, transformation → Feed Exporter
8. Feed Exporter → writes CSV/JSON to disk
Architectural patterns and design principles
• Modular separation:
– Crawling logic (Spiders)
– Request/response manipulation (Middlewares)
– Data modeling (Items)
– Data processing/export (Pipelines, Feed Exporter)
• Asynchronous engine (Twisted event loop) with configurable concurrency
• Command-line driven, decoupled configuration via settings.py
• Reusable components (Item Loaders, XPath selectors)