How johnserf-seed/f2 works
Project type and purpose
• F2 is a Python‑based, multi‑platform content downloader and data‑processing library with both a CLI and programmable API interfaces.
File structure
• Top‑level “f2” folder: the application code, subdivided into:
• apps/…: one subpackage per supported platform, each with api.py, cli.py, crawler.py, dl.py, db.py, filter.py, handler.py, model.py, utils.py
• cli/: core command processor and console front‑end
• conf/: YAML configuration files and conf‑manager utilities
• crawlers/: base_crawler.py defines HTTP & WebSocket request framework
• dl/: base_downloader.py for standardized download workflows
• db/: base_db.py for storage (sqlite via aiosqlite)
• exceptions/: custom error types
• i18n/, languages/: localization support
• log/: centralized logger
• utils/: shared helpers (singleton, signal, JSON filters, abogus/xbogus algorithms)
• tests/ and f2/apps/*/test: unit and integration tests
• docs/ & .vitepress/: website‑generation for end‑user documentation
README for additional insights
• Confirms CLI usage, configuration files (app.yaml, conf.yaml, defaults.yaml), and extensibility points.
• Describes features (fetch_user_profile, fetch_one_video, fetch_live_im, etc.), notification integration (Bark), and advanced guides.
a. Main components
• User (CLI or Python API client)
• CLI Front‑end (f2/cli) or Programmatic Entry (f2/__main__.py)
• Command Processor & Dispatcher
• Platform‑Specific Module Loader (f2/apps/*)
• Shared Services:
• Configuration Manager (f2/conf & f2/utils/conf_manager.py)
• Crawler Layer (f2/crawlers/base_crawler.py)
• Downloader Layer (f2/dl/base_downloader.py)
• Data Storage (f2/db/base_db.py + aiosqlite)
• Logging (f2/log/logger.py)
• Internationalization (f2/i18n/translator.py + language packs)
• Utilities & Algorithms (XBogus, ABogus managers, token/signature generators)
• External Services:
• Social‑media HTTP APIs (Douyin, TikTok, Twitter, WeiBo)
• WebSocket endpoints for live‑stream IM
• Notification Service (Bark HTTP API)
b. Relationships & interactions
• User → CLI/API → Command Processor
• Command Processor → Loads platform module → Invokes api/crawler/dl/db/filter in sequence
• Crawler → HTTP/WebSocket call → External platform → returns raw data
• Filter/Model → transforms raw data → Data Storage (optional) → File System (downloads)
• Logging and i18n intercept at each layer
• Config Manager supplies credentials, endpoints, feature flags
c. Architectural patterns & principles
• Plugin‑style “apps” for each platform (extensible, isolated code)
• Shared base classes for crawler/downloader/db to enforce consistency
• Separation of concerns: CLI vs core logic vs persistence vs networking
• Asynchronous I/O for network and file operations (httpx, websockets, aiosqlite)
• Dependency injection via config manager
d. Relevant technologies/libraries
• click or rich for CLI console
• httpx for HTTP requests
• websockets for live‑streaming
• protobuf for live‑cast messages
• pydantic for data models
• aiosqlite for persistence
• PyExecJS & cryptography for signature algorithms