How teamwiseflow/wiseflow works
Project Overview & Main Components Identification
• Project Type & Purpose: Wiseflow is an LLM-native information harvesting and summarization tool. It automatically crawls various web pages, extracts target information using language model wrappers and scraping agents, and stores/processes the data with an integrated PocketBase database. There is also a dashboard component to allow management and report generation.
• Main Components to Include:
• Web Crawling & Scraping Layer (in “core/scrapers” and agents in “core/agents”)
• LLM Integration Layer (in “core/llms” wrapping OpenAI and Litellm APIs)
• Core Processing/Task Orchestration (in “core/general_process.py”, “core/run_task.py”, etc.)
• Utility Functions (in “core/utils” for search functions and API integrations, e.g., exa_search, pb_api, zhipu_search)
• Database Layer: PocketBase integration (pb directory and related migration scripts)
• Dashboard / UI Layer: A separate “dashboard” directory (with backend.py, main.py, and additional crawler utilities) to interact with retrieved information and present reports
• External Integrations: “weixin_mp” for Weixin messaging integration and Docker-related files for containerization
Analyzing the File Structure for Architectural Insights
• The “core” directory hosts the main processing logic:
• “agents” directory: contains modules to gather actions, prompts, and insights.
• “llms” directory: encapsulates wrappers for different large language model APIs, highlighting the abstraction over different LLM providers.
• “scrapers” directory: contains web crawling logic and specialized modules (e.g., customer_crawl4ai_md_generator) for data extraction.
• “utils” directory: holds general utility functions (search, API wrappers) supporting other components.
• Shell scripts (run.sh, docker_entrypoint.sh, windows_run.py) serve as orchestration for execution, potentially in different environments.
• The “dashboard” is somewhat separate and designed for a more UI-driven or report-based interaction with the data.
• The “pb” folder indicates database migration scripts and configuration for PocketBase, showing that the data store is auto-managed.
• Testing directories hold comprehensive tests and sample report extractions.
Final Detailed Steps
1. Data Acquisition Layer (Scraping & Agents)
2. Processing Layer (LLM Integration + Core Orchestration)
3. Persistence Layer (PocketBase Database)
4. Presentation Layer (Dashboard & UI Components)
5. External Integration & Deployment (Docker, Weixin API, Environment Config)
• Expand each block showing internal modules (for example, break “Data Acquisition” into “core/scrapers” and “core/agents”).
• Review with deployment and integration points noted in the README (e.g., instructions for Docker deployment, .env configuration, and PocketBase installation).