# johnserf-seed/f2 architecture

> Architecture diagram of the GitHub repository johnserf-seed/f2, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-22). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/johnserf-seed/f2
- Repository: https://github.com/johnserf-seed/f2

## Overview

##### Project type and purpose
- F2 is a Python‑based, multi‑platform content downloader and data‑processing library with both a CLI and programmable API interfaces.

##### File structure
- Top‑level “f2” folder: the application code, subdivided into:
- apps/…: one subpackage per supported platform, each with api.py, cli.py, crawler.py, dl.py, db.py, filter.py, handler.py, model.py, utils.py
- cli/: core command processor and console front‑end
- conf/: YAML configuration files and conf‑manager utilities
- crawlers/: base_crawler.py defines HTTP & WebSocket request framework
- dl/: base_downloader.py for standardized download workflows
- db/: base_db.py for storage (sqlite via aiosqlite)
- exceptions/: custom error types
- i18n/, languages/: localization support
- log/: centralized logger
- utils/: shared helpers (singleton, signal, JSON filters, abogus/xbogus algorithms)
- tests/ and f2/apps/*/test: unit and integration tests
- docs/ & .vitepress/: website‑generation for end‑user documentation

##### README for additional insights
- Confirms CLI usage, configuration files (app.yaml, conf.yaml, defaults.yaml), and extensibility points.
- Describes features (fetch_user_profile, fetch_one_video, fetch_live_im, etc.), notification integration (Bark), and advanced guides.

a. Main components
- User (CLI or Python API client)
- CLI Front‑end (f2/cli) or Programmatic Entry (f2/__main__.py)
- Command Processor & Dispatcher
- Platform‑Specific Module Loader (f2/apps/*)
- Shared Services:
- Configuration Manager (f2/conf & f2/utils/conf_manager.py)
- Crawler Layer (f2/crawlers/base_crawler.py)
- Downloader Layer (f2/dl/base_downloader.py)
- Data Storage (f2/db/base_db.py + aiosqlite)
- Logging (f2/log/logger.py)
- Internationalization (f2/i18n/translator.py + language packs)
- Utilities & Algorithms (XBogus, ABogus managers, token/signature generators)
- External Services:
- Social‑media HTTP APIs (Douyin, TikTok, Twitter, WeiBo)
- WebSocket endpoints for live‑stream IM
- Notification Service (Bark HTTP API)
b. Relationships & interactions
- User → CLI/API → Command Processor
- Command Processor → Loads platform module → Invokes api/crawler/dl/db/filter in sequence
- Crawler → HTTP/WebSocket call → External platform → returns raw data
- Filter/Model → transforms raw data → Data Storage (optional) → File System (downloads)
- Logging and i18n intercept at each layer
- Config Manager supplies credentials, endpoints, feature flags
c. Architectural patterns & principles
- Plugin‑style “apps” for each platform (extensible, isolated code)
- Shared base classes for crawler/downloader/db to enforce consistency
- Separation of concerns: CLI vs core logic vs persistence vs networking
- Asynchronous I/O for network and file operations (httpx, websockets, aiosqlite)
- Dependency injection via config manager
d. Relevant technologies/libraries
- click or rich for CLI console
- httpx for HTTP requests
- websockets for live‑streaming
- protobuf for live‑cast messages
- pydantic for data models
- aiosqlite for persistence
- PyExecJS & cryptography for signature algorithms

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% User Layer
    subgraph "User Layer"
        User["User\n(CLI or Python API)"]:::user
    end

    %% Presentation Layer
    subgraph "Presentation Layer"
        CLI["CLI Front-end"]:::presentation
        APIEntry["Programmatic Entry"]:::presentation
        CmdProc["Command Processor & Dispatcher"]:::presentation
    end

    %% Application Core
    subgraph "Application Core"
        AppLoader["Platform Module Loader"]:::core
        ConfigMgr["Configuration Manager"]:::core
        Crawler["BaseCrawler"]:::core
        Downloader["BaseDownloader"]:::core
        DB["BaseDB"]:::core
        Logger["Logging Service"]:::core
        I18n["i18n Translator"]:::core
        Helpers["Core Helpers"]:::core
        subgraph "Shared Utilities & Algorithms"
            Singleton["_singleton"]:::core
            Signal["_signal"]:::core
            ABogus["ABogus Algorithm"]:::core
            XBogus["XBogus Algorithm"]:::core
            JSONF["JSON Filter"]:::core
        end
    end

    %% Plugins
    subgraph "Platform Apps"
        Douyin["Douyin App"]:::plugin
        TikTok["TikTok App"]:::plugin
        Twitter["Twitter App"]:::plugin
        WeiBo["WeiBo App"]:::plugin
        Bark["Bark Notification App"]:::plugin
    end

    %% Infrastructure Layer
    subgraph "Infrastructure Layer"
        subgraph "Configuration Files"
            AppY["app.yaml"]:::infra
            ConfY["conf.yaml"]:::infra
            DefY["defaults.yaml"]:::infra
        end
        Database["SQLite (aiosqlite)"]:::infra
        FileSystem["File System"]:::infra
    end

    %% External Services
    subgraph "External Services"
        SMAPI["Social Media HTTP APIs"]:::external
        WS["WebSocket Endpoints"]:::external
        Notification["Bark HTTP API"]:::external
    end

    %% Connections
    User --> CLI
    User --> APIEntry
    CLI --> CmdProc
    APIEntry --> CmdProc
    CmdProc --> AppLoader
    AppLoader --> Douyin
    AppLoader --> TikTok
    AppLoader --> Twitter
    AppLoader --> WeiBo
    AppLoader --> Bark
    CmdProc --> ConfigMgr
    ConfigMgr --> AppY
    ConfigMgr --> ConfY
    ConfigMgr --> DefY
    CmdProc --> Logger
    CmdProc --> I18n
    ConfigMgr --> I18n
    AppLoader --> Crawler
    Crawler --> SMAPI
    Crawler --> WS
    Crawler --> Logger
    Crawler --> I18n
    Crawler --> Singleton
    Crawler --> Signal
    Crawler --> ABogus
    Crawler --> XBogus
    Crawler --> JSONF
    Crawler --> Downloader
    Downloader --> FileSystem
    Downloader --> DB
    Downloader --> Logger
    Downloader --> Singleton
    DB --> Database
    DB --> Logger
    DB --> Singleton
    Notification --> Notification

    %% Click Events
    click CLI "https://github.com/johnserf-seed/f2/blob/main/f2/cli/cli_console.py"
    click CmdProc "https://github.com/johnserf-seed/f2/blob/main/f2/cli/cli_commands.py"
    click APIEntry "https://github.com/johnserf-seed/f2/blob/main/f2/__main__.py"
    click AppLoader "https://github.com/johnserf-seed/f2/blob/main/f2/apps/__apps__.py"
    click Douyin "https://github.com/johnserf-seed/f2/tree/main/f2/apps/douyin/"
    click TikTok "https://github.com/johnserf-seed/f2/tree/main/f2/apps/tiktok/"
    click Twitter "https://github.com/johnserf-seed/f2/tree/main/f2/apps/twitter/"
    click WeiBo "https://github.com/johnserf-seed/f2/tree/main/f2/apps/weibo/"
    click Bark "https://github.com/johnserf-seed/f2/tree/main/f2/apps/bark/"
    click ConfigMgr "https://github.com/johnserf-seed/f2/blob/main/f2/utils/conf_manager.py"
    click AppY "https://github.com/johnserf-seed/f2/blob/main/f2/conf/app.yaml"
    click ConfY "https://github.com/johnserf-seed/f2/blob/main/f2/conf/conf.yaml"
    click DefY "https://github.com/johnserf-seed/f2/blob/main/f2/conf/defaults.yaml"
    click Crawler "https://github.com/johnserf-seed/f2/blob/main/f2/crawlers/base_crawler.py"
    click Downloader "https://github.com/johnserf-seed/f2/blob/main/f2/dl/base_downloader.py"
    click DB "https://github.com/johnserf-seed/f2/blob/main/f2/db/base_db.py"
    click Logger "https://github.com/johnserf-seed/f2/blob/main/f2/log/logger.py"
    click I18n "https://github.com/johnserf-seed/f2/blob/main/f2/i18n/translator.py"
    click Singleton "https://github.com/johnserf-seed/f2/blob/main/f2/utils/_singleton.py"
    click Signal "https://github.com/johnserf-seed/f2/blob/main/f2/utils/_signal.py"
    click ABogus "https://github.com/johnserf-seed/f2/blob/main/f2/utils/abogus.py"
    click XBogus "https://github.com/johnserf-seed/f2/blob/main/f2/utils/xbogus.py"
    click JSONF "https://github.com/johnserf-seed/f2/blob/main/f2/utils/json_filter.py"

    %% Styles
    classDef user fill:#afeeee,stroke:#333,stroke-width:1px
    classDef presentation fill:#add8e6,stroke:#333,stroke-width:1px
    classDef core fill:#90ee90,stroke:#333,stroke-width:1px
    classDef plugin fill:#98fb98,stroke:#333,stroke-width:1px
    classDef infra fill:#ffdead,stroke:#333,stroke-width:1px
    classDef external fill:#dda0dd,stroke:#333,stroke-width:1px
```
