# openaispace/ai-trend-publish architecture

> Architecture diagram of the GitHub repository openaispace/ai-trend-publish, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-31). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/openaispace/ai-trend-publish
- Repository: https://github.com/openaispace/ai-trend-publish

## Overview

Based on both the file tree and the README, TrendPublish is a backend system built using Deno and TypeScript that focuses on trend discovery and automated content publishing (especially for Weixin/微信公众号). Its purpose is to scrape data from multiple sources (e.g., Twitter, websites via FireCrawl), process this data through AI-based summarization and ranking, convert content (using Markdown converters and templates), and finally publish articles automatically, while also providing notifications for task status and errors.

##### Main Components
- Scraping & Data Source Modules: Capture the modules under src/modules/scrapers and src/data-sources. These components are responsible for retrieving raw content from external sources.
- AI & Content Processing:
- Markdown Converter: Include a clear component for the Markdown conversion (with detailed sub-components for the various renderers in src/modules/md-converter) that translates source content into formatted text.
- Template Rendering & Publishing:
- Notification Services: Outline the notify modules (e.g., Dingding and Bark) that are used to push alerts and status messages.
- Workflow Orchestration: Represent scheduled tasks and workflows from src/controllers/cron.ts, src/works, and services (such as weixin-article.workflow.ts). This flows the overall process—from data collection to publication.
- Configuration & Database:
- Config Manager: Include a component for the configuration management (src/utils/config) that loads .env and other configuration details.
- Database: Show the local database integration managed via Drizzle in the drizzle folder and src/db, used for storing state, publication history, or content data.
- External AI/Image Providers: Mark the LLM and image generation providers (from src/providers) as external integration points that offer AI services and image generation capabilities.

##### Relationships and Interactions
- Content Transformation: From content processing flow, the output is then passed into the Markdown Converter. Once converted, it moves into the Template Rendering modules for final formatting.
- Publication: After rendering, the Publisher module pushes the formatted articles to the Weixin API (or potentially other publication channels).
- Notifications: In parallel or on specific outcomes (success or error), the Notification Services are called to alert users/administrators.
- Workflow Coordination: Show orchestration by a central Workflow Manager that triggers these steps at scheduled intervals (using the cron controller or workflows from src/works).
- Configuration & External Integration: Always denote that components such as AI services, image generators, and even the database are configured centrally through the config module.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data Ingestion Layer
    subgraph "Data Ingestion Layer"
        EDS["External Data Sources"]:::external
        subgraph "Scraping Modules"
            TS["Twitter Scraper"]:::internal
            FS["FireCrawl Scraper"]:::internal
            HS["HelloGithub Scraper"]:::internal
        end
    end

    %% Content Processing Layer
    subgraph "Content Processing Layer"
        AS["AI Summarizer"]:::processing
        CR["Content Ranker"]:::processing
        CRP["Content Ranker Prompt"]:::processing
        MC["Markdown Converter"]:::processing
    end

    %% Content Publication Layer
    subgraph "Content Publication Layer"
        TR["Template Renderer"]:::publication
        CP["Content Publisher"]:::publication
    end

    %% Notification Services
    subgraph "Notification Services"
        DNG["Dingding Notify"]:::notification
        BARK["Bark Notify"]:::notification
    end

    %% Workflow Orchestration
    subgraph "Workflow Orchestration"
        CRON["Cron Controller"]:::workflow
        WORK["Workflows"]:::workflow
        WAX["Weixin Article Workflow"]:::workflow
    end

    %% Supporting Components
    CM["Config Manager"]:::config
    DB["Database"]:::database
    LLM["LLM Providers"]:::provider
    IMG["Image Generation Providers"]:::provider

    %% Data Flow from External Sources to Scrapers
    EDS -->|"fetches"| TS
    EDS -->|"fetches"| FS
    EDS -->|"fetches"| HS

    %% Scraping Modules feed into AI Processing
    TS -->|"sends raw data"| AS
    FS -->|"sends raw data"| AS
    HS -->|"sends raw data"| AS

    TS -->|"sends raw data"| CR
    FS -->|"sends raw data"| CR
    HS -->|"sends raw data"| CR

    %% Content Ranker uses a prompt module
    CR -->|"uses"| CRP

    %% AI Processing outputs to Markdown Conversion
    AS -->|"summarizes"| MC
    CR -->|"ranks"| MC

    %% Markdown Conversion to Template Rendering
    MC -->|"converts to markdown"| TR

    %% Template Rendering to Content Publisher
    TR -->|"formats content"| CP

    %% Content Publisher publishes and logs to Database
    CP -->|"publishes content"| DB

    %% Notification flow from Publication
    CP -->|"alerts"| DNG
    CP -->|"alerts"| BARK

    %% Workflow Orchestration triggers key modules (dotted arrows)
    CRON -.-> TS
    CRON -.-> AS
    CRON -.-> MC
    CRON -.-> TR
    CRON -.-> CP

    %% Configuration influence
    CM -->|"configures"| AS
    CM -->|"configures"| CR
    CM -->|"configures"| MC
    CM -->|"configures"| TR
    CM -->|"configures"| CP
    CM -->|"configures"| DB
    CM -->|"configures"| LLM
    CM -->|"configures"| IMG

    %% External Providers integration
    LLM -->|"integrated with"| AS
    LLM -->|"integrated with"| CR
    IMG -->|"integrated with"| CP

    %% Workflow internal connections
    WORK --> WAX

    %% Click Events
    click EDS "https://github.com/openaispace/ai-trend-publish/blob/main/src/data-sources/getDataSources.ts"
    click TS "https://github.com/openaispace/ai-trend-publish/blob/main/src/modules/scrapers/twitter.scraper.ts"
    click FS "https://github.com/openaispace/ai-trend-publish/blob/main/src/modules/scrapers/fireCrawl.scraper.ts"
    click HS "https://github.com/openaispace/ai-trend-publish/blob/main/src/modules/scrapers/hellogithub.scraper.ts"
    click AS "https://github.com/openaispace/ai-trend-publish/blob/main/src/modules/summarizer/ai.summarizer.ts"
    click CR "https://github.com/openaispace/ai-trend-publish/blob/main/src/modules/content-rank/ai.content-ranker.ts"
    click CRP "https://github.com/openaispace/ai-trend-publish/blob/main/src/prompts/content-ranker.prompt.ts"
    click MC "https://github.com/openaispace/ai-trend-publish/tree/main/src/modules/md-converter"
    click TR "https://github.com/openaispace/ai-trend-publish/tree/main/src/modules/render/weixin"
    click CP "https://github.com/openaispace/ai-trend-publish/blob/main/src/modules/publishers/weixin.publisher.ts"
    click DNG "https://github.com/openaispace/ai-trend-publish/blob/main/src/modules/notify/dingding.notify.ts"
    click BARK "https://github.com/openaispace/ai-trend-publish/blob/main/src/modules/notify/bark.notify.ts"
    click CRON "https://github.com/openaispace/ai-trend-publish/blob/main/src/controllers/cron.ts"
    click WORK "https://github.com/openaispace/ai-trend-publish/blob/main/src/works/index.ts"
    click WAX "https://github.com/openaispace/ai-trend-publish/blob/main/src/services/weixin-article.workflow.ts"
    click CM "https://github.com/openaispace/ai-trend-publish/blob/main/src/utils/config/config-manager.ts"
    click DB "https://github.com/openaispace/ai-trend-publish/blob/main/src/db/db.ts"
    click LLM "https://github.com/openaispace/ai-trend-publish/blob/main/src/providers/llm/llm-factory.ts"
    click IMG "https://github.com/openaispace/ai-trend-publish/tree/main/src/providers/image-gen"

    %% Style Classes
    classDef external fill:#f9e79f,stroke:#000,stroke-width:2px;
    classDef internal fill:#aed6f1,stroke:#000,stroke-width:2px;
    classDef processing fill:#abebc6,stroke:#000,stroke-width:2px;
    classDef publication fill:#f9c79f,stroke:#000,stroke-width:2px;
    classDef notification fill:#f5b7b1,stroke:#000,stroke-width:2px;
    classDef workflow fill:#d2b4de,stroke:#000,stroke-width:2px;
    classDef config fill:#fcf3cf,stroke:#000,stroke-width:2px;
    classDef database fill:#aed6f1,stroke:#000,stroke-width:2px;
    classDef provider fill:#f5cba7,stroke:#000,stroke-width:2px;
```
