# just-ads/ai-goofish architecture

> Architecture diagram of the GitHub repository just-ads/ai-goofish, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-11-24). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/just-ads/ai-goofish
- Repository: https://github.com/just-ads/ai-goofish

## Overview

##### Project type and purpose
- This is a full-stack monitoring and scraping application called “Goofish Monitor,” built in Python.
- Purpose: continuously scrape (via Playwright) and evaluate items on the Xianyu marketplace, store results, expose a REST API (FastAPI) for task management and querying, and serve a web UI for operators. It also integrates AI-based product evaluation and multiple notification channels.

##### File structure
Top level:
- start.py / start_spider.py – entrypoints
- Dockerfile, docker-compose.yml, .env.example – containerization
- requirements.txt – Python deps

src/ – backend code
- config.py – global settings and environment variables
- server/
– server.py: FastAPI app exposing task & result endpoints and serving static assets
– scheduler.py: orchestrates and dispatches scraping tasks
- spider/
– spider.py: core Playwright-based crawler
– parsers.py: HTML parsing logic for item & seller data
- task/
– task.py: task definition & lifecycle
– result.py: result storage/querying (JSONL file-based)
- agent/
– product_evaluator.py: AI evaluation pipeline (title, seller, item analysis)
– client.py: AI service client wrapper
- notify/
– notify_manager.py: unified interface for notification channels (ntfy, Gotify, WeChat)
– gotify.py, ntfy.py, wechat_service.py: channel implementations
- utils/
– file_operator.py: file I/O helper
– utils.py: common utility functions

resources/static/ – built front-end assets (JS/CSS) served by FastAPI
webui/ – original front-end source (e.g. React/Vue)

##### README for additional insights
- Describes a three-step AI evaluation pipeline.
- Uses JWT for web UI authentication.
- Task scheduling supports parallelism, anti-bot handling, and result queries.
- Deployment via Docker Compose.

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Frontend Layer
    subgraph "Frontend"
        direction TB
        WebUI["Web UI Source\n(webui/)"]:::external
        Browser["User Browser"]:::external
        StaticAssets["Static Assets\n(resources/static/)"]:::internal
        WebUI -->|Build assets| StaticAssets
        Browser -->|load HTML| StaticAssets
        Browser -->|login, CRUD tasks,\nview results| FastAPI
    end

    %% Backend Container
    subgraph "Backend Container (Docker)" 
        direction TB
        EntrypointStart["Entry Point\nstart.py"]:::internal
        Config["Config\nsrc/config.py"]:::internal
        FastAPI["FastAPI Server\nsrc/server/server.py"]:::internal
        EntrypointStart -->|reads| Config
        EntrypointStart -->|starts| FastAPI

        subgraph "Scheduler Module"
            direction TB
            Scheduler["Scheduler\nsrc/server/scheduler.py"]:::internal
        end

        subgraph "Spider Module"
            direction TB
            SpiderEntrypoint["Spider Entrypoint\nstart_spider.py"]:::internal
            Spider["Spider\nsrc/spider/spider.py"]:::internal
            Parsers["Parsers\nsrc/spider/parsers.py"]:::internal
            SpiderEntrypoint -->|launch| Spider
            Spider -->|parse HTML| Parsers
        end

        subgraph "AI Agent"
            direction TB
            Agent["ProductEvaluator\nsrc/agent/product_evaluator.py"]:::internal
            Client["AI Client\nsrc/agent/client.py"]:::internal
            Agent -->|calls| Client
        end

        subgraph "Notification Manager"
            direction TB
            NotifyMgr["Notify Manager\nsrc/notify/notify_manger.py"]:::internal
            Gotify["Gotify\nsrc/notify/gotify.py"]:::internal
            Ntfy["Ntfy\nsrc/notify/ntfy.py"]:::internal
            WeChat["WeChat\nsrc/notify/wechat_service.py"]:::internal
            NotifyMgr --> Gotify
            NotifyMgr --> Ntfy
            NotifyMgr --> WeChat
        end

        subgraph "Storage"
            direction TB
            TaskDef["Task Definitions\nsrc/task/task.py"]:::internal
            ResultStore["Results Storage\nsrc/task/result.py"]:::storage
        end

        subgraph "Utilities"
            direction TB
            FileUtil["File I/O\nsrc/utils/file_operator.py"]:::internal
            Utils["Common Utils\nsrc/utils/utils.py"]:::internal
        end

        FastAPI -->|enqueue task| Scheduler
        Scheduler -->|spawn job| SpiderEntrypoint
        Spider -->|write results| ResultStore
        Scheduler -->|invoke evaluation| Agent
        Agent -->|store eval| ResultStore
        FastAPI -->|query results| ResultStore
        FastAPI -->|trigger notifications| NotifyMgr
    end

    %% Containerization
    subgraph "Deployment"
        direction TB
        Dockerfile["Dockerfile"]:::external
        Compose["docker-compose.yml"]:::external
        Env[".env.example"]:::external
        Req["requirements.txt"]:::external
    end

    %% External Services
    subgraph "External Services"
        direction TB
        PlaywrightEngine["Playwright Engine"]:::thirdparty
        AIAPI["AI API (OpenAI)"]:::thirdparty
        NotifyServices["Notification Endpoints"]:::thirdparty
    end

    Spider -->|controls| PlaywrightEngine
    Agent -->|calls| AIAPI
    NotifyMgr -->|sends notifications| NotifyServices

    %% Class Definitions
    classDef internal fill:#bbdefb,stroke:#2196f3,color:#0d47a1;
    classDef storage fill:#c8e6c9,stroke:#388e3c,color:#1b5e20;
    classDef external fill:#ffe0b2,stroke:#fb8c00,color:#e65100;
    classDef thirdparty fill:#e1bee7,stroke:#8e24aa,color:#4a148c;

    %% Click Events
    click WebUI "https://github.com/just-ads/ai-goofish/tree/master/webui/"
    click StaticAssets "https://github.com/just-ads/ai-goofish/tree/master/resources/static/"
    click EntrypointStart "https://github.com/just-ads/ai-goofish/blob/master/start.py"
    click Config "https://github.com/just-ads/ai-goofish/blob/master/src/config.py"
    click FastAPI "https://github.com/just-ads/ai-goofish/blob/master/src/server/server.py"
    click Scheduler "https://github.com/just-ads/ai-goofish/blob/master/src/server/scheduler.py"
    click Spider "https://github.com/just-ads/ai-goofish/blob/master/src/spider/spider.py"
    click Parsers "https://github.com/just-ads/ai-goofish/blob/master/src/spider/parsers.py"
    click SpiderEntrypoint "https://github.com/just-ads/ai-goofish/blob/master/start_spider.py"
    click Agent "https://github.com/just-ads/ai-goofish/blob/master/src/agent/product_evaluator.py"
    click Client "https://github.com/just-ads/ai-goofish/blob/master/src/agent/client.py"
    click NotifyMgr "https://github.com/just-ads/ai-goofish/blob/master/src/notify/notify_manger.py"
    click Gotify "https://github.com/just-ads/ai-goofish/blob/master/src/notify/gotify.py"
    click Ntfy "https://github.com/just-ads/ai-goofish/blob/master/src/notify/ntfy.py"
    click WeChat "https://github.com/just-ads/ai-goofish/blob/master/src/notify/wechat_service.py"
    click TaskDef "https://github.com/just-ads/ai-goofish/blob/master/src/task/task.py"
    click ResultStore "https://github.com/just-ads/ai-goofish/blob/master/src/task/result.py"
    click FileUtil "https://github.com/just-ads/ai-goofish/blob/master/src/utils/file_operator.py"
    click Utils "https://github.com/just-ads/ai-goofish/blob/master/src/utils/utils.py"
    click Dockerfile "https://github.com/just-ads/ai-goofish/tree/master/Dockerfile"
    click Compose "https://github.com/just-ads/ai-goofish/blob/master/docker-compose.yml"
    click Env "https://github.com/just-ads/ai-goofish/blob/master/.env.example"
    click Req "https://github.com/just-ads/ai-goofish/blob/master/requirements.txt"
```
