How datawhores/of-scraper works
Project Type & Purpose
• This is a command-line scraping tool that interacts with OnlyFans’ APIs (with a heavy emphasis on authentication, media downloading, metadata processing, and filtering). It is designed to support bulk actions and a variety of filtering options.
• The README and file structure confirm that it is not a typical full-stack web app but rather a CLI tool with several independent modules (data, commands, download actions, logging, configuration, etc.).
High-Level Architectural Components
a. CLI & Command Parsing:
• Entry point: The “ofscraper/__main__.py” and the various commands (found in “ofscraper/commands” and “ofscraper/utils/args/parse”), which handle user input, command-line parameters, and option parsing.
• Interactive prompts are also managed under “ofscraper/prompts” for things such as keybindings, prompt groups, and conversion utilities.
b. Auth & Session Management:
• Authentication utilities are in “ofscraper/utils/auth” and “ofscraper/classes/sessionmanager”. They encapsulate user authentication to OnlyFans, session creation, and management needed for access (given OnlyFans requires a subscription/logged-in context).
c. API & Data Handling:
• A significant portion of the code is dedicated to API interactions, organized in “ofscraper/data/api”. This module handles endpoints like profiles, timelines, subscriptions, and different media types.
• The “ofscraper/data/models” supports the mapping of the data models that the scraper uses internally.
d. Download & Action Execution:
• The scraping functionality, including downloading and “liking” actions, is found in “ofscraper/commands/scraper/actions.” These files illustrate different download managers (e.g., “downloadmanager.py”, “alt_download.py”) and helper utilities (e.g., “chunk.py”, “retries.py”) to reliably download content.
• In addition, modules for ‘like’ actions and metadata processing are also present.
e. Data Persistence & Database Operations:
• The “ofscraper/db” directory shows scripts for database backup, merge, difference calculation, and various read/write operations ensuring metadata consistency and duplicate check across runs.
• This is crucial for the tool’s ability to check and prevent redundant downloads.
f. Logging, Progress, and Runtime Utilities:
• Directories like “ofscraper/utils/logs”, “ofscraper/utils/live”, and parts of the “ofscraper/utils/config” module handle logging, real-time progress updates (with rich text displays), and configuration file management.
g. Auxiliary Components:
• Additional functionalities include filtering logic (“ofscraper/filters”), command helpers (“ofscraper/classes/table” for potentially formatting output in a table), and various other helper modules found in “ofscraper/utils” (paths, constants, formatting, etc.).
Relationships & Interactions
• The user interacts with the CLI front-end which routes commands to the appropriate module.
• The CLI calls into command modules (like “scraper”, “db”, “metadata”), which in turn use helper libraries/modules to access and process OnlyFans data via the API submodules.
• Authentication and session managers act as intermediaries ensuring that every API call uses valid credentials.
• Download managers coordinate with retry mechanisms, progress trackers, and logging utilities to report status.
• The configuration and logging utilities serve cross-cutting concerns for nearly every component (from command parsing to API requests).
• Data persistence and database folders ensure stateful operations (e.g., caching downloads, metadata discrepancy checks).
Architectural Patterns & Technologies
• The design leverages a modular structure with clear separation of concerns: CLI routing, API interfacing, action handling, and persistence are separated.
• An “MVC-like” or layered approach is visible where the commands (controllers) use business logic (models/services) and view components (CLI/table outputs).
• Use of retry and multi-threading utilities for robust download actions suggests that concurrency and fault tolerance are integrated into the design.
• Docker support is also evident (with Dockerfile, docker-compose.yml) for containerized builds/deployment.