How just-ads/ai-goofish works
Project type and purpose
• This is a full-stack monitoring and scraping application called “Goofish Monitor,” built in Python.
• Purpose: continuously scrape (via Playwright) and evaluate items on the Xianyu marketplace, store results, expose a REST API (FastAPI) for task management and querying, and serve a web UI for operators. It also integrates AI-based product evaluation and multiple notification channels.
File structure
Top level:
• start.py / start_spider.py – entrypoints
• Dockerfile, docker-compose.yml, .env.example – containerization
• requirements.txt – Python deps
src/ – backend code
• config.py – global settings and environment variables
• server/
– server.py: FastAPI app exposing task & result endpoints and serving static assets
– scheduler.py: orchestrates and dispatches scraping tasks
• spider/
– spider.py: core Playwright-based crawler
– parsers.py: HTML parsing logic for item & seller data
• task/
– task.py: task definition & lifecycle
– result.py: result storage/querying (JSONL file-based)
• agent/
– product_evaluator.py: AI evaluation pipeline (title, seller, item analysis)
– client.py: AI service client wrapper
• notify/
– notify_manager.py: unified interface for notification channels (ntfy, Gotify, WeChat)
– gotify.py, ntfy.py, wechat_service.py: channel implementations
• utils/
– file_operator.py: file I/O helper
– utils.py: common utility functions
resources/static/ – built front-end assets (JS/CSS) served by FastAPI
webui/ – original front-end source (e.g. React/Vue)
README for additional insights
• Describes a three-step AI evaluation pipeline.
• Uses JWT for web UI authentication.
• Task scheduling supports parallelism, anti-bot handling, and result queries.
• Deployment via Docker Compose.