How kevinzg/facebook-scraper works
Project type and purpose
• This is a Python library and CLI tool (“facebook-scraper”) for scraping public Facebook pages, profiles, groups, posts, comments, and related metadata without using the official API.
• It provides a set of high-level functions (get_posts, get_profile, get_group_info, write_posts_to_csv) and a CLI entry point (facebook_scraper/__main__.py).
File structure
• Top-level files:
– pyproject.toml, requirements.txt, requirements-dev.txt: dependency definitions
– README.md: documentation and usage examples
– LICENSE, .gitignore, .flake8: project metadata/config
• Core package directory: facebook_scraper
– __main__.py: CLI entrypoint
– facebook_scraper.py: high-level API (get_posts, get_profile, get_group_info, write_posts_to_csv)
– page_iterators.py: logic to paginate and fetch pages of HTML
– extractors.py: parsing HTML and extracting post/profile/group data
– utils.py: cookie handling, HTTP helpers, retry logic
– constants.py, fb_types.py: type definitions and constant values
– exceptions.py: custom exception classes
• tests directory: unit tests and recorded HTTP cassettes for offline testing
README insights
• Describes command-line usage and Python API, optional parameters, and examples of data structures returned.
• Mentions optional youtube-dl integration for video extraction.
• Highlights output formats (generator of Python dicts or CSV files).
Main components
a. CLI Layer
• facebook-scraper CLI (invokes __main__.py)
• Parses command-line args, calls library API
b. API Layer
• High-level functions: get_posts, get_profile, get_group_info, write_posts_to_csv
• Exposes Python interface (module facebook_scraper.facebook_scraper)
c. Scraper Core
• Page Iterator (page_iterators.py): constructs paginated HTTP requests
• HTTP Client (requests library, session configured via utils.py)
• Authentication/Cookies Manager (utils.py)
• Extractors (extractors.py): HTML parsing (BeautifulSoup or lxml), CSS selectors, JSON blobs
• Data Models (fb_types.py): define the schema for posts, comments, profiles
• Exceptions (exceptions.py): error handling, retries
d. Optional Modules
• youtube-dl integration for video media extraction
e. Output Layer
• In-memory Python objects (generators, dicts)
• CSV Writer (embedded in write_posts_to_csv)
• Progress reporting (tqdm progress bars)
f. External Services
• Facebook web endpoints (HTTP/HTTPS)
• Optional local filesystem (cookie files, CSV files)
Relationships and interactions
• CLI → API layer: CLI parses args and dispatches to get_posts or write_posts_to_csv
• API layer → Scraper core: calls PageIterator to fetch pages, then passes raw HTML to Extractors
• Scraper core → HTTP Client: issues GET requests to Facebook, handles cookies/auth
• HTTP Client ↔ Facebook: network calls, responses (HTML, JSON snippets)
• Extractors → Data Models: build Python dicts or objects from parsed content
• API layer → Output: yield generators or write to CSV on disk
Architectural patterns and design principles
• Layered architecture: Presentation (CLI), API, Domain (scraper core), Infrastructure (HTTP client, filesystem)
• Separation of concerns: pagination logic vs. extraction logic vs. HTTP/cookie management
• Modular design: extractors can be extended to add new fields or support new Facebook page types
• Dependency injection: utilities for cookie and session management are centralized in utils
• Testability: tests are decoupled via HTTP cassettes for deterministic responses
Essential elements to capture
• Clear labels for each module and external system
• Notes on optional paths (e.g., when youtube-dl is enabled)
• Representation of configuration sources (cookies file, credentials tuple)