How 2977094657/bilibilihistoryfetcher works
Project Identification
• Purpose: This is a backend system that fetches, processes, analyzes, and visualizes a Bilibili user’s viewing history. It supports data collection, various analytical reports (annual summaries, video, and image downloads), automation (scheduler tasks), and even audio-to-text via GPU-accelerated modules.
• Type: A data-driven backend service that exposes API endpoints (via the “routers” directory) and interacts with utility scripts. It is coupled with a separate front-end project (BiliHistoryFrontend) but primarily focuses on API and automation tasks.
Analyzing the File Structure
• Configuration and Setup:
• Files in the “config” directory (config.yaml, scheduler_config.yaml, etc.) indicate centralized configuration for B站 API authentication, email notifications, DeepSeek API, server settings, and scheduler tasks.
• Files like CUDA_SETUP.md and install_dependencies.py confirm dependency and hardware configuration (CUDA/CPU variants, PyTorch, etc.).
• Core Application:
• “main.py” is the entry point, likely setting up the server and initializing routes.
• “app.spec”, “app_full.spec”, “app_lite.spec”, and “build.py” are used for packaging the app into standalone executables, with different build options (Lite vs Full versions).
• API Routing:
• The “routers” folder contains individual modules for various endpoints (e.g., analysis, audio_to_text, login, download, scheduler, etc.). Each of these corresponds to a specific functionality exposed via an HTTP API.
• Utility & Data Processing:
• The “scripts” directory hosts multiple scripts handling tasks like data extraction, cleaning, analysis (e.g., analyze_bilibili_history.py, comment_fetcher.py, scheduler-related scripts), which can either be invoked manually or scheduled.
• Dependency & Environment Management:
• Files like requirements.txt and install_dependencies.py ensure that all required Python packages are installed along with special handling for GPU acceleration.
Key Components
a. Main Components:
• Client / Front-end: (External) The BiliHistoryFrontend that consumes the API.
• API Server: The core HTTP server launched via “main.py” that binds all API endpoints.
• Routers/Endpoints: Each module in “routers” (for fetching history, exporting data, audio-to-text conversion, etc.) should be depicted as separate service nodes or grouped based on functionality.
• Task Scheduler: Represented by both the scheduler endpoints in “routers/scheduler.py” and the scripts managing scheduled tasks (in “scripts/scheduler*.py”).
• Database Access: Although not a full-fledged DB component in itself, include “sql_statements_mysql.py” and “sql_statements_sqlite.py” to indicate interactions with the chosen database (SQLite or MySQL).
• External Integrations:
• Email Service for notifications (configures SMTP settings).
• DeepSeek AI API for advanced summary/analysis.
• FFmpeg for video processing.
• PyTorch and faster-whisper for audio to text conversion.
b. Relationships & Interactions:
• Data flows from user requests (front-end or automated scheduler) to API endpoints, which then process or delegate tasks to the appropriate data processing modules.
• Configuration files provide settings that are read by almost every module (API server, scheduler, and dependent processing scripts).
• The packaging/build process (build.py, app_xxx.spec) wraps the entire functionality into executable bundles.
• Request/Response cycles between Frontend and API Server.
• Data flow from API endpoints to utility scripts/modules for processing.
• Scheduler invocations triggering individual processing tasks and subsequently sending email notifications or visual outputs.
c. Architectural Patterns/Design Principles:
• Separation of concerns: Clearly dividing API routing (routers), configuration (config files), scheduled/background tasks (scripts), and packaging.
• Modularity: Organizing each endpoint or task in its individual module/script enables easy testing and maintenance.
• Extensibility: The design supports additional external integrations (DeepSeek, FFmpeg, CUDA acceleration) without major structural changes, as seen in the configuration and dedicated integration files.
d. Relevant Technologies & Frameworks:
• Python 3.10+, FastAPI (implied by the auto-generated docs at /docs and the modular routers).
• SQLite/MySQL (as possible persistence layers).
• PyInstaller for application packaging.
• CUDA/PyTorch/faster-whisper for audio processing.
• Email (SMTP) configuration for notifications.
Specific Elements
• “API Server (main.py)”, “Routers (analysis, login, etc.)”, “Scheduler”, “Data Processing Scripts”, “Config Manager”, “Packaging Module”
• Data flow from Front-end → API Server → Routers → Scripts/External Services.
• Scheduler triggering periodic tasks that feed into data processing flows and then to email notifications.
• Details such as supported technologies (e.g., CUDA-enabled modules, PyTorch dependency) and the build process (Lite vs Full executables).