# rugantio/fbcrawl architecture

> Architecture diagram of the GitHub repository rugantio/fbcrawl, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-08-01). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/rugantio/fbcrawl
- Repository: https://github.com/rugantio/fbcrawl

## Overview

##### Project type and purpose
- This is a Python web crawler for Facebook pages and posts, built on the Scrapy framework.
- Two main spiders: one for posts (“fb”) and one for comments.

##### File structure
- Top level:
– scrapy.cfg (Scrapy project configuration)
– runner_facebook.sh (example shell wrapper)
- fbcrawl package:
– settings.py (framework settings, pipelines, concurrency, robots.txt)
– middlewares.py (custom request/response hooks, login checkpoint handling)
– items.py (declaration of scraped data fields)
– pipelines.py (date-filter pipeline, CSV/JSON exporters)
– spiders/ (directory containing fbcrawl.py, comments.py, plus auxiliary spiders for events and profiles)

##### Extract architecture details from the README
- The Scrapy Engine orchestrates:
– Scheduler (queues Requests)
– Downloader & Downloader Middlewares (HTTPS requests to mbasic.facebook.com)
– Spider components (parse logic using lxml/XPath)
– Item Pipeline (post-processing, filtering, exporting)
- External dependency: Facebook’s mobile HTML interface (mbasic.facebook.com)
- Output: CSV or JSON via Scrapy’s Feed Exporter

##### Main components
a. Entrypoint
– CLI/runner_facebook.sh invoking “scrapy crawl fb|comments” with arguments.
b. Configuration
– scrapy.cfg + settings.py feeding into the Scrapy Engine.
– Scheduler
– Downloader (with custom Downloader Middlewares)
– Spiders (fb spider and comments spider)
– Item Pipeline (FbcrawlPipeline)
d. External service
– mbasic.facebook.com (Facebook mobile HTML interface)
e. Data storage
– CSV/JSON via Scrapy Feed Exporter
f. Supporting libraries
– Python3, Scrapy, Twisted, lxml, XPath

##### Relationships and data flow
1. User → CLI (runner_facebook.sh or scrapyd command)
2. CLI → Scrapy Engine (loads settings, instantiates scheduler, downloader, spiders, pipelines)
3. Scrapy Engine → Scheduler (enqueue initial Request)
4. Scheduler → Downloader → External Service (HTTP GET)
5. External Service → Downloader → Downloader Middlewares (login, cookies) → Spider.parse
6. Spider.parse → produces new Requests (cycle back to scheduler) and Items
7. Items → Item Pipeline (FbcrawlPipeline) → filtering, transformation → Feed Exporter
8. Feed Exporter → writes CSV/JSON to disk

##### Architectural patterns and design principles
- Modular separation:
– Crawling logic (Spiders)
– Request/response manipulation (Middlewares)
– Data modeling (Items)
– Data processing/export (Pipelines, Feed Exporter)
- Asynchronous engine (Twisted event loop) with configurable concurrency
- Command-line driven, decoupled configuration via settings.py
- Reusable components (Item Loaders, XPath selectors)

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Entry and Configuration
    User["User"] 
    Runner["runner_facebook.sh"]:::project
    Config["scrapy.cfg"]:::project
    Settings["settings.py"]:::project

    User -->|"runs"| Runner
    Runner -->|"invokes Scrapy"| Engine

    Config -->|"defines project"| Engine
    Settings -->|"configures engine"| Engine

    %% Scrapy Engine and Core Components
    subgraph "Scrapy Engine"
        direction TB
        Scheduler["Scheduler"]:::core
        Downloader["Downloader"]:::core
        Middlewares["Downloader Middlewares\n(login & cookies)"]:::project
        subgraph "Spiders"
            direction TB
            PostSpider["fbcrawl.py (Posts Spider)"]:::project
            CommentSpider["comments.py (Comments Spider)"]:::project
            EventsSpider["events.py"]:::project
            ProfilesSpider["profiles.py"]:::project
        end
        Pipeline["FbcrawlPipeline\n(filtering & export)"]:::project
    end

    Engine["Scrapy Engine"]:::core

    Engine -->|"schedules requests"| Scheduler
    Scheduler -->|"downloads"| Downloader
    Downloader -->|"process request"| Middlewares
    Middlewares -->|"passes response"| PostSpider
    Middlewares -->|"passes response"| CommentSpider
    Middlewares -->|"passes response"| EventsSpider
    Middlewares -->|"passes response"| ProfilesSpider

    PostSpider -->|"new Requests"| Scheduler
    CommentSpider -->|"new Requests"| Scheduler
    EventsSpider -->|"new Requests"| Scheduler
    ProfilesSpider -->|"new Requests"| Scheduler

    PostSpider -->|"items"| Pipeline
    CommentSpider -->|"items"| Pipeline
    EventsSpider -->|"items"| Pipeline
    ProfilesSpider -->|"items"| Pipeline

    %% External Service and Storage
    External["mbasic.facebook.com"]:::ext
    Storage["CSV/JSON Files"]:::ext

    Downloader -->|"HTTP GET"| External
    External -->|"HTML response"| Downloader
    Pipeline -->|"sends to exporter"| Storage

    %% Click Events
    click Runner "https://github.com/rugantio/fbcrawl/blob/master/runner_facebook.sh"
    click Config "https://github.com/rugantio/fbcrawl/blob/master/scrapy.cfg"
    click Settings "https://github.com/rugantio/fbcrawl/blob/master/fbcrawl/settings.py"
    click Items "https://github.com/rugantio/fbcrawl/blob/master/fbcrawl/items.py"
    click Middlewares "https://github.com/rugantio/fbcrawl/blob/master/fbcrawl/middlewares.py"
    click Pipeline "https://github.com/rugantio/fbcrawl/blob/master/fbcrawl/pipelines.py"
    click PostSpider "https://github.com/rugantio/fbcrawl/blob/master/fbcrawl/spiders/fbcrawl.py"
    click CommentSpider "https://github.com/rugantio/fbcrawl/blob/master/fbcrawl/spiders/comments.py"
    click EventsSpider "https://github.com/rugantio/fbcrawl/blob/master/fbcrawl/spiders/events.py"
    click ProfilesSpider "https://github.com/rugantio/fbcrawl/blob/master/fbcrawl/spiders/profiles.py"

    %% Styles
    classDef core fill:#ADD8E6,stroke:#000,stroke-width:1px
    classDef project fill:#90EE90,stroke:#000,stroke-width:1px
    classDef ext fill:#FFA500,stroke:#000,stroke-width:1px
```
