yz302/radar

Purpose and entry points

TrendRadar is a Python news-hotspot aggregation and monitoring system. It fetches configured platform hot lists plus RSS/Atom feeds, filters and ranks items, persists history, generates reports, and distributes notifications. It also exposes stored news to AI clients through MCP.

Primary runtime entry points:

• Main crawler/report pipeline: trendradar/__main__.py

• MCP server, usable over STDIO or HTTP: mcp_server/server.py

• Container operational wrapper/web-server manager: docker/manage.py

Application orchestration and configuration

The application package is organized around a central execution context and configuration/data-loading layer. Configuration controls enabled sources, reporting mode, storage backend, notification channels, AI, and environment-variable overrides. Keyword groups and global filters are separate text configuration.

Paths: trendradar/context.py, trendradar/core/config.py, trendradar/core/loader.py, config/config.yaml, config/frequency_words.txt.

Control boundary: configuration is loaded before collection and is shared by crawling, storage, reporting, notification, and optional AI stages.

Collection and normalization

Platform collection is an HTTP-fetching crawler that obtains hot-list data from the external newsnow API/service. RSS/Atom has a parallel fetch/parse path. Collected items become normalized internal news records for subsequent history-aware processing.

Paths: trendradar/crawler/fetcher.py, trendradar/crawler/rss/fetcher.py, trendradar/crawler/rss/parser.py.

External boundary: newsnow supplies multi-platform trending data; configured RSS/Atom endpoints are independently fetched. Network failures and source-specific formats are isolated in this layer.

Analysis, filtering, and trend state

Core analysis applies keyword groups, required/excluded terms, regex matching, frequency-based scoring, ranking, URL normalization, and detection of new versus previously seen items. Historical records support daily, current-list, and incremental reporting semantics.

Paths: trendradar/core/analyzer.py, trendradar/core/frequency.py, trendradar/core/data.py, trendradar/utils/url.py.

The resulting categorized data is the main contract between ingestion/storage and presentation/delivery.

Storage and retention

Storage is abstracted behind a manager and supports local SQLite by default plus remote S3-compatible object storage for stateless environments. SQLite schemas include separate hot-list and RSS structures; daily databases are represented under output/news/.

Paths: trendradar/storage/manager.py, trendradar/storage/local.py, trendradar/storage/remote.py, trendradar/storage/schema.sql, trendradar/storage/rss_schema.sql.

Infrastructure boundary: Docker/local runs persist mounted local output; GitHub Actions can use S3-compatible providers such as Cloudflare R2, OSS, or COS. Retention and synchronization are storage concerns rather than crawler concerns.

Report, AI enrichment, and notification delivery

Report generation converts analyzed records into text/HTML/RSS-oriented presentation. Optional AI analysis and translation call models through LiteLLM using configurable prompts, then embed results into the report. Notification dispatch formats, splits, and sends payloads to channel adapters.

Paths: trendradar/report/generator.py, trendradar/report/html.py, trendradar/ai/client.py, trendradar/ai/analyzer.py, trendradar/notification/dispatcher.py, trendradar/notification/senders.py.

External services include LiteLLM-supported LLM providers and webhook/SMTP-based channels: WeCom, Feishu, DingTalk, Telegram, ntfy, Bark, Slack, generic webhooks, and email. HTML output is also published as static index.html.

MCP analytics service

The MCP subsystem is a separate read-oriented interface over persisted TrendRadar data. Its server registers query, search, analytics, configuration, synchronization, and system tools. Services parse arguments, access/caches stored databases, and return a consistent result structure.

Paths: mcp_server/server.py, mcp_server/services/data_service.py, mcp_server/services/parser_service.py, mcp_server/services/cache_service.py, mcp_server/tools/analytics.py, mcp_server/tools/data_query.py.

It requires existing output/ data; it does not itself crawl live news. Docker can run it separately with read-only config/output volumes, typically HTTP on localhost port 3333.

Deployment and automation

The same Python code supports local execution, Docker Compose, and scheduled GitHub Actions. Docker separates crawler/notification and MCP images; Actions schedule collection and provide a stateless runtime, making remote storage architecturally important.

Paths: docker/Dockerfile, docker/Dockerfile.mcp, docker/docker-compose.yml, .github/workflows/crawler.yml, .github/workflows/clean-crawler.yml.