# kevinzg/facebook-scraper architecture

> Architecture diagram of the GitHub repository kevinzg/facebook-scraper, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-07-07). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/kevinzg/facebook-scraper
- Repository: https://github.com/kevinzg/facebook-scraper

## Overview

##### Project type and purpose
- This is a Python library and CLI tool (“facebook-scraper”) for scraping public Facebook pages, profiles, groups, posts, comments, and related metadata without using the official API.
- It provides a set of high-level functions (get_posts, get_profile, get_group_info, write_posts_to_csv) and a CLI entry point (facebook_scraper/__main__.py).

##### File structure
- Top-level files:
– pyproject.toml, requirements.txt, requirements-dev.txt: dependency definitions
– README.md: documentation and usage examples
– LICENSE, .gitignore, .flake8: project metadata/config
- Core package directory: facebook_scraper
– __main__.py: CLI entrypoint
– facebook_scraper.py: high-level API (get_posts, get_profile, get_group_info, write_posts_to_csv)
– page_iterators.py: logic to paginate and fetch pages of HTML
– extractors.py: parsing HTML and extracting post/profile/group data
– utils.py: cookie handling, HTTP helpers, retry logic
– constants.py, fb_types.py: type definitions and constant values
– exceptions.py: custom exception classes
- tests directory: unit tests and recorded HTTP cassettes for offline testing

##### README insights
- Describes command-line usage and Python API, optional parameters, and examples of data structures returned.
- Mentions optional youtube-dl integration for video extraction.
- Highlights output formats (generator of Python dicts or CSV files).

##### Main components
a. CLI Layer
- facebook-scraper CLI (invokes __main__.py)
- Parses command-line args, calls library API
b. API Layer
- High-level functions: get_posts, get_profile, get_group_info, write_posts_to_csv
- Exposes Python interface (module facebook_scraper.facebook_scraper)
c. Scraper Core
- Page Iterator (page_iterators.py): constructs paginated HTTP requests
- HTTP Client (requests library, session configured via utils.py)
- Authentication/Cookies Manager (utils.py)
- Extractors (extractors.py): HTML parsing (BeautifulSoup or lxml), CSS selectors, JSON blobs
- Data Models (fb_types.py): define the schema for posts, comments, profiles
- Exceptions (exceptions.py): error handling, retries
d. Optional Modules
- youtube-dl integration for video media extraction
e. Output Layer
- In-memory Python objects (generators, dicts)
- CSV Writer (embedded in write_posts_to_csv)
- Progress reporting (tqdm progress bars)
f. External Services
- Facebook web endpoints (HTTP/HTTPS)
- Optional local filesystem (cookie files, CSV files)

##### Relationships and interactions
- CLI → API layer: CLI parses args and dispatches to get_posts or write_posts_to_csv
- API layer → Scraper core: calls PageIterator to fetch pages, then passes raw HTML to Extractors
- Scraper core → HTTP Client: issues GET requests to Facebook, handles cookies/auth
- HTTP Client ↔ Facebook: network calls, responses (HTML, JSON snippets)
- Extractors → Data Models: build Python dicts or objects from parsed content
- API layer → Output: yield generators or write to CSV on disk

##### Architectural patterns and design principles
- Layered architecture: Presentation (CLI), API, Domain (scraper core), Infrastructure (HTTP client, filesystem)
- Separation of concerns: pagination logic vs. extraction logic vs. HTTP/cookie management
- Modular design: extractors can be extended to add new fields or support new Facebook page types
- Dependency injection: utilities for cookie and session management are centralized in utils
- Testability: tests are decoupled via HTTP cassettes for deterministic responses

##### Essential elements to capture
- Clear labels for each module and external system
- Notes on optional paths (e.g., when youtube-dl is enabled)
- Representation of configuration sources (cookies file, credentials tuple)

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% CLI Layer
    CLI["CLI: facebook-scraper"]:::cli
    click CLI "https://github.com/kevinzg/facebook-scraper/blob/master/facebook_scraper/__main__.py"

    %% API Layer
    API["API Layer\n(facebook_scraper.py)"]:::api
    click API "https://github.com/kevinzg/facebook-scraper/blob/master/facebook_scraper/facebook_scraper.py"

    %% Scraper Core
    subgraph "Scraper Core"
        PI["PageIterator"]:::internal
        click PI "https://github.com/kevinzg/facebook-scraper/blob/master/facebook_scraper/page_iterators.py"

        HC["HTTP Client &\nCookie Manager"]:::internal
        click HC "https://github.com/kevinzg/facebook-scraper/blob/master/facebook_scraper/utils.py"

        EX["Extractors"]:::internal
        click EX "https://github.com/kevinzg/facebook-scraper/blob/master/facebook_scraper/extractors.py"

        DM["Data Models"]:::internal
        click DM "https://github.com/kevinzg/facebook-scraper/blob/master/facebook_scraper/fb_types.py"

        CO["Constants"]:::internal
        click CO "https://github.com/kevinzg/facebook-scraper/blob/master/facebook_scraper/constants.py"

        ER["Exceptions"]:::internal
        click ER "https://github.com/kevinzg/facebook-scraper/blob/master/facebook_scraper/exceptions.py"

        YD["youtube-dl Integration"]:::externalLib
    end

    %% Output Layer
    Output["Output Layer\n(CSV Writer & Progress)"]:::output

    %% External Services
    FB["Facebook Web Endpoints"]:::externalService
    FS["Filesystem\n(cookie & CSV files)"]:::externalService

    %% Tests
    subgraph "Tests & Cassettes"
        T1["test_get_posts.py"]:::tests
        click T1 "https://github.com/kevinzg/facebook-scraper/blob/master/tests/test_get_posts.py"
        T2["test_parse_date.py"]:::tests
        click T2 "https://github.com/kevinzg/facebook-scraper/blob/master/tests/test_parse_date.py"
        T3["test_parse_duration.py"]:::tests
        click T3 "https://github.com/kevinzg/facebook-scraper/blob/master/tests/test_parse_duration.py"
        TC["cassettes"]:::tests
        click TC "https://github.com/kevinzg/facebook-scraper/tree/master/tests/cassettes"
    end

    %% Relationships
    CLI -->|"parse args"| API
    API -->|"invoke pagination"| PI
    PI -->|"HTTP requests"| HC
    HC -->|"GET requests"| FB
    FB -->|"HTML/JSON responses"| HC
    HC -->|"raw HTML"| EX
    EX -->|"parsed dicts"| DM
    DM -->|"models"| API
    API -->|"generator of dicts"| Output
    Output -->|"write CSV"| FS
    HC -->|"read/write cookies"| FS
    EX -->|"video hooks"| YD

    %% Styling
    classDef cli fill:#b3e5fc,stroke:#0277bd,color:#000
    classDef api fill:#bbdefb,stroke:#0d47a1,color:#000
    classDef internal fill:#e3f2fd,stroke:#1a237e,color:#000
    classDef externalService fill:#ffe0b2,stroke:#ef6c00,color:#000
    classDef externalLib fill:#c8e6c9,stroke:#2e7d32,color:#000
    classDef output fill:#c5e1a5,stroke:#558b2f,color:#000
    classDef tests fill:#f8bbd0,stroke:#ad1457,color:#000
```
