# teamwiseflow/wiseflow architecture

> Architecture diagram of the GitHub repository teamwiseflow/wiseflow, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-03). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/teamwiseflow/wiseflow
- Repository: https://github.com/teamwiseflow/wiseflow

## Overview

##### Project Overview & Main Components Identification
- Project Type & Purpose: Wiseflow is an LLM-native information harvesting and summarization tool. It automatically crawls various web pages, extracts target information using language model wrappers and scraping agents, and stores/processes the data with an integrated PocketBase database. There is also a dashboard component to allow management and report generation.
- Main Components to Include:
- Web Crawling & Scraping Layer (in “core/scrapers” and agents in “core/agents”)
- LLM Integration Layer (in “core/llms” wrapping OpenAI and Litellm APIs)
- Core Processing/Task Orchestration (in “core/general_process.py”, “core/run_task.py”, etc.)
- Utility Functions (in “core/utils” for search functions and API integrations, e.g., exa_search, pb_api, zhipu_search)
- Database Layer: PocketBase integration (pb directory and related migration scripts)
- Dashboard / UI Layer: A separate “dashboard” directory (with backend.py, main.py, and additional crawler utilities) to interact with retrieved information and present reports
- External Integrations: “weixin_mp” for Weixin messaging integration and Docker-related files for containerization

##### Analyzing the File Structure for Architectural Insights
- The “core” directory hosts the main processing logic:
- “agents” directory: contains modules to gather actions, prompts, and insights.
- “llms” directory: encapsulates wrappers for different large language model APIs, highlighting the abstraction over different LLM providers.
- “scrapers” directory: contains web crawling logic and specialized modules (e.g., customer_crawl4ai_md_generator) for data extraction.
- “utils” directory: holds general utility functions (search, API wrappers) supporting other components.
- Shell scripts (run.sh, docker_entrypoint.sh, windows_run.py) serve as orchestration for execution, potentially in different environments.
- The “dashboard” is somewhat separate and designed for a more UI-driven or report-based interaction with the data.
- The “pb” folder indicates database migration scripts and configuration for PocketBase, showing that the data store is auto-managed.
- Testing directories hold comprehensive tests and sample report extractions.

##### Final Detailed Steps
1. Data Acquisition Layer (Scraping & Agents)
2. Processing Layer (LLM Integration + Core Orchestration)
3. Persistence Layer (PocketBase Database)
4. Presentation Layer (Dashboard & UI Components)
5. External Integration & Deployment (Docker, Weixin API, Environment Config)
- Expand each block showing internal modules (for example, break “Data Acquisition” into “core/scrapers” and “core/agents”).
- Review with deployment and integration points noted in the README (e.g., instructions for Docker deployment, .env configuration, and PocketBase installation).

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data Acquisition Layer
    subgraph "Data Acquisition Layer"
        scrapers["Scrapers"]:::ingestion
        agents["Agents"]:::ingestion
    end

    %% Processing Layer
    subgraph "Processing Layer"
        llm["LLM Integration"]:::processing
        openai["OpenAI Wrapper"]:::processing
        liteLLM["LiteLLM Wrapper"]:::processing
        orchestrator["Core Orchestrator"]:::processing
        process_general["general_process.py"]:::processing
        process_run_task["run_task.py"]:::processing
        process_run_sh["run.sh"]:::processing
        process_docker_entry["docker_entrypoint.sh"]:::processing
        utils["Utility Functions"]:::utils
        exa_search["exa_search.py"]:::utils
        pb_api["pb_api.py"]:::utils
        zhipu_search["zhipu_search.py"]:::utils
        general_utils["general_utils.py"]:::utils
    end

    %% Persistence Layer
    subgraph "Persistence Layer"
        pocketbase["PocketBase Database"]:::database
        migrations["Migrations"]:::database
        install_sh["Install PocketBase (sh)"]:::database
        install_ps1["Install PocketBase (ps1)"]:::database
    end

    %% Presentation Layer
    subgraph "Presentation Layer"
        dashboard["Dashboard UI"]:::dashboard
        backend["backend.py"]:::dashboard
        main_py["main.py"]:::dashboard
        get_report["get_report.py"]:::dashboard
    end

    %% External Integration
    subgraph "External Integration"
        weixin["Weixin Messaging"]:::external
    end

    %% Deployment/Containerization
    subgraph "Deployment/Containerization"
        dockerCompose["Docker Compose"]:::deployment
        envDocker["Env Docker"]:::deployment
    end

    %% Data Flow Connections
    scrapers -->|"data"| llm
    agents -->|"data"| llm
    llm -->|"calls"| openai
    llm -->|"calls"| liteLLM
    llm -->|"processed_data"| orchestrator
    orchestrator -->|"uses"| process_general
    orchestrator -->|"uses"| process_run_task
    orchestrator -->|"uses"| process_run_sh
    orchestrator -->|"uses"| process_docker_entry
    utils -->|"helpers"| exa_search
    utils -->|"helpers"| pb_api
    utils -->|"helpers"| zhipu_search
    utils -->|"helpers"| general_utils
    orchestrator -->|"utilizes"| utils
    orchestrator -->|"stores"| pocketbase
    pocketbase -->|"feeds"| dashboard
    dashboard -->|"queries"| pocketbase
    orchestrator -->|"alerts"| weixin
    dockerCompose -->|"deploys"| orchestrator
    dockerCompose -->|"deploys"| pocketbase
    dockerCompose -->|"deploys"| dashboard
    envDocker -->|"configures"| orchestrator

    %% Click Events
    click scrapers "https://github.com/teamwiseflow/wiseflow/tree/master/core/scrapers"
    click agents "https://github.com/teamwiseflow/wiseflow/tree/master/core/agents"
    click llm "https://github.com/teamwiseflow/wiseflow/tree/master/core/llms"
    click openai "https://github.com/teamwiseflow/wiseflow/blob/master/core/llms/openai_wrapper.py"
    click liteLLM "https://github.com/teamwiseflow/wiseflow/blob/master/core/llms/litellm_wrapper.py"
    click process_general "https://github.com/teamwiseflow/wiseflow/blob/master/core/general_process.py"
    click process_run_task "https://github.com/teamwiseflow/wiseflow/blob/master/core/run_task.py"
    click process_run_sh "https://github.com/teamwiseflow/wiseflow/blob/master/core/run.sh"
    click process_docker_entry "https://github.com/teamwiseflow/wiseflow/blob/master/core/docker_entrypoint.sh"
    click utils "https://github.com/teamwiseflow/wiseflow/tree/master/core/utils"
    click exa_search "https://github.com/teamwiseflow/wiseflow/blob/master/core/utils/exa_search.py"
    click pb_api "https://github.com/teamwiseflow/wiseflow/blob/master/core/utils/pb_api.py"
    click zhipu_search "https://github.com/teamwiseflow/wiseflow/blob/master/core/utils/zhipu_search.py"
    click general_utils "https://github.com/teamwiseflow/wiseflow/blob/master/core/utils/general_utils.py"
    click pocketbase "https://github.com/teamwiseflow/wiseflow/tree/master/pb"
    click migrations "https://github.com/teamwiseflow/wiseflow/tree/master/pb/pb_migrations"
    click install_sh "https://github.com/teamwiseflow/wiseflow/blob/master/pb/install_pocketbase.sh"
    click install_ps1 "https://github.com/teamwiseflow/wiseflow/blob/master/pb/install_pocketbase.ps1"
    click dashboard "https://github.com/teamwiseflow/wiseflow/tree/master/dashboard"
    click backend "https://github.com/teamwiseflow/wiseflow/blob/master/dashboard/backend.py"
    click main_py "https://github.com/teamwiseflow/wiseflow/blob/master/dashboard/main.py"
    click get_report "https://github.com/teamwiseflow/wiseflow/blob/master/dashboard/get_report.py"
    click weixin "https://github.com/teamwiseflow/wiseflow/tree/master/weixin_mp"
    click dockerCompose "https://github.com/teamwiseflow/wiseflow/blob/master/docker-compose.yml"
    click envDocker "https://github.com/teamwiseflow/wiseflow/tree/master/env_docker"

    %% Styles
    classDef ingestion fill:#cce5ff,stroke:#003366,stroke-width:2px;
    classDef processing fill:#d4edda,stroke:#155724,stroke-width:2px;
    classDef utils fill:#fff3cd,stroke:#856404,stroke-width:2px;
    classDef database fill:#f8d7da,stroke:#721c24,stroke-width:2px;
    classDef dashboard fill:#d1ecf1,stroke:#0c5460,stroke-width:2px;
    classDef external fill:#e2e3e5,stroke:#6c757d,stroke-width:2px;
    classDef deployment fill:#fefefe,stroke:#343a40,stroke-width:2px;
```
