# datawhores/of-scraper architecture

> Architecture diagram of the GitHub repository datawhores/of-scraper, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-30). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/datawhores/of-scraper
- Repository: https://github.com/datawhores/of-scraper

## Overview

##### Project Type & Purpose
- This is a command-line scraping tool that interacts with OnlyFans’ APIs (with a heavy emphasis on authentication, media downloading, metadata processing, and filtering). It is designed to support bulk actions and a variety of filtering options.
- The README and file structure confirm that it is not a typical full-stack web app but rather a CLI tool with several independent modules (data, commands, download actions, logging, configuration, etc.).

##### High-Level Architectural Components
a. CLI & Command Parsing:
- Entry point: The “ofscraper/__main__.py” and the various commands (found in “ofscraper/commands” and “ofscraper/utils/args/parse”), which handle user input, command-line parameters, and option parsing.
- Interactive prompts are also managed under “ofscraper/prompts” for things such as keybindings, prompt groups, and conversion utilities.

b. Auth & Session Management:
- Authentication utilities are in “ofscraper/utils/auth” and “ofscraper/classes/sessionmanager”. They encapsulate user authentication to OnlyFans, session creation, and management needed for access (given OnlyFans requires a subscription/logged-in context).

c. API & Data Handling:
- A significant portion of the code is dedicated to API interactions, organized in “ofscraper/data/api”. This module handles endpoints like profiles, timelines, subscriptions, and different media types.
- The “ofscraper/data/models” supports the mapping of the data models that the scraper uses internally.

d. Download & Action Execution:
- The scraping functionality, including downloading and “liking” actions, is found in “ofscraper/commands/scraper/actions.” These files illustrate different download managers (e.g., “downloadmanager.py”, “alt_download.py”) and helper utilities (e.g., “chunk.py”, “retries.py”) to reliably download content.
- In addition, modules for ‘like’ actions and metadata processing are also present.

e. Data Persistence & Database Operations:
- The “ofscraper/db” directory shows scripts for database backup, merge, difference calculation, and various read/write operations ensuring metadata consistency and duplicate check across runs.
- This is crucial for the tool’s ability to check and prevent redundant downloads.

f. Logging, Progress, and Runtime Utilities:
- Directories like “ofscraper/utils/logs”, “ofscraper/utils/live”, and parts of the “ofscraper/utils/config” module handle logging, real-time progress updates (with rich text displays), and configuration file management.

g. Auxiliary Components:
- Additional functionalities include filtering logic (“ofscraper/filters”), command helpers (“ofscraper/classes/table” for potentially formatting output in a table), and various other helper modules found in “ofscraper/utils” (paths, constants, formatting, etc.).

##### Relationships & Interactions
- The user interacts with the CLI front-end which routes commands to the appropriate module.
- The CLI calls into command modules (like “scraper”, “db”, “metadata”), which in turn use helper libraries/modules to access and process OnlyFans data via the API submodules.
- Authentication and session managers act as intermediaries ensuring that every API call uses valid credentials.
- Download managers coordinate with retry mechanisms, progress trackers, and logging utilities to report status.
- The configuration and logging utilities serve cross-cutting concerns for nearly every component (from command parsing to API requests).
- Data persistence and database folders ensure stateful operations (e.g., caching downloads, metadata discrepancy checks).

##### Architectural Patterns & Technologies
- The design leverages a modular structure with clear separation of concerns: CLI routing, API interfacing, action handling, and persistence are separated.
- An “MVC-like” or layered approach is visible where the commands (controllers) use business logic (models/services) and view components (CLI/table outputs).
- Use of retry and multi-threading utilities for robust download actions suggests that concurrency and fault tolerance are integrated into the design.
- Docker support is also evident (with Dockerfile, docker-compose.yml) for containerized builds/deployment.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Main Components
    CLI["CLI & Command Parser"]:::core
    Auth["Authentication & Session Management"]:::service
    API["API & Data Handling"]:::service
    Models["Data Models"]:::service
    Download["Download & Action Execution"]:::process
    DB["Data Persistence & DB Operations"]:::database
    Logs["Logging, Progress & Runtime Utilities"]:::utility
    Config["Configuration Management"]:::utility
    Aux["Auxiliary Components (Filtering, Prompts, Helpers)"]:::helper
    External["External OnlyFans API"]:::external
    Docker["Docker Deployment"]:::deployment

    %% Interactions
    CLI -->|"triggers"| Auth
    CLI -->|"sendsCommands"| Download
    CLI -->|"invokesAPI"| API
    CLI -->|"usesAux"| Aux
    Auth -->|"authenticates"| API
    API -->|"utilizes"| Models
    API -->|"fetchesData"| External
    Download -->|"downloadsAndProcesses"| API
    Download -->|"logsProgress"| Logs
    DB -->|"persistsData"| API
    Logs -->|"reportsStatus"| CLI
    Config -->|"configuresCLI"| CLI
    Config -->|"configuresLogs"| Logs
    Docker -->|"hosts"| CLI

    %% Click Events for CLI & Command Parsing
    click CLI "https://github.com/datawhores/of-scraper/blob/main/ofscraper/__main__.py"
    click CLI "https://github.com/datawhores/of-scraper/tree/main/ofscraper/commands"
    click CLI "https://github.com/datawhores/of-scraper/tree/main/ofscraper/utils/args/parse"

    %% Click Events for Authentication & Session Management
    click Auth "https://github.com/datawhores/of-scraper/tree/main/ofscraper/utils/auth"
    click Auth "https://github.com/datawhores/of-scraper/tree/main/ofscraper/classes/sessionmanager"

    %% Click Events for API & Data Handling
    click API "https://github.com/datawhores/of-scraper/tree/main/ofscraper/data/api"
    click Models "https://github.com/datawhores/of-scraper/tree/main/ofscraper/data/models"

    %% Click Event for Download & Action Execution
    click Download "https://github.com/datawhores/of-scraper/tree/main/ofscraper/commands/scraper/actions"

    %% Click Event for Data Persistence & Database Operations
    click DB "https://github.com/datawhores/of-scraper/tree/main/ofscraper/db"

    %% Click Events for Logging, Progress & Runtime Utilities
    click Logs "https://github.com/datawhores/of-scraper/tree/main/ofscraper/utils/logs"
    click Logs "https://github.com/datawhores/of-scraper/tree/main/ofscraper/utils/live"

    %% Click Event for Configuration Management
    click Config "https://github.com/datawhores/of-scraper/tree/main/ofscraper/utils/config"

    %% Click Events for Auxiliary Components
    click Aux "https://github.com/datawhores/of-scraper/tree/main/ofscraper/filters"
    click Aux "https://github.com/datawhores/of-scraper/tree/main/ofscraper/prompts"

    %% Styles
    classDef core fill:#FFD700,stroke:#333,stroke-width:2px;
    classDef service fill:#87CEEB,stroke:#333,stroke-width:2px;
    classDef process fill:#90EE90,stroke:#333,stroke-width:2px;
    classDef database fill:#FFA07A,stroke:#333,stroke-width:2px;
    classDef utility fill:#C0C0C0,stroke:#333,stroke-width:2px;
    classDef helper fill:#F08080,stroke:#333,stroke-width:2px;
    classDef external fill:#FF69B4,stroke:#333,stroke-width:2px;
    classDef deployment fill:#9370DB,stroke:#333,stroke-width:2px;
```
