# 666ghj/bettafish architecture

> Architecture diagram of the GitHub repository 666ghj/bettafish, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-04-06). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/666ghj/bettafish
- Repository: https://github.com/666ghj/bettafish

## Overview

1. **Repository purpose and top-level shape**  
   BettaFish is a multi-agent opinion-analysis system. The root app orchestrates a full pipeline from data acquisition and analysis to report generation. The repository is split into four main runtime domains: crawling/data collection, three specialized analysis engines, a forum-style coordination layer, and a report engine that converts analysis outputs into final HTML/PDF deliverables.

2. **User-facing entry points**  
   The main web entry is `app.py`, backed by a Flask UI in `templates/` and `static/`. There are also standalone Streamlit entry points under `SingleEngineApp/` for running each analysis engine independently. For report-only workflows, `report_engine_only.py` and the `regenerate_latest_*.py` scripts bypass the analysis step and rebuild outputs from saved artifacts.

3. **Three analysis engines with a shared architecture pattern**  
   `QueryEngine/`, `MediaEngine/`, and `InsightEngine/` follow the same internal structure: an `agent.py` orchestrator, `llms/` abstraction, `nodes/` for stepwise reasoning, `tools/` for external actions, `prompts/` for prompt templates, `state/` for runtime state, and `utils/` for configuration and text handling. This consistent layout makes them separate but parallel subsystems, each specializing in a different data source or analysis style.

4. **What each analysis engine is responsible for**  
   `QueryEngine` focuses on broad web/news search and topic discovery.  
   `MediaEngine` focuses on multimodal content analysis, especially social/video/image-oriented public content.  
   `InsightEngine` focuses on private database mining and deeper structured analysis, including search over stored records and sentiment analysis integration.  
   Architecturally, these engines are peer services: they collect different evidence streams, then produce intermediate markdown/state outputs consumed later by reporting.

5. **ForumEngine as the coordination boundary**  
   `ForumEngine/` is the cross-agent collaboration layer. It monitors agent activity and uses an LLM “host” to simulate a moderated forum discussion. The repository’s core idea is that agents do not only work independently; they also exchange findings through this shared coordination mechanism, which influences subsequent search/reflection cycles. The `utils/forum_reader.py` helper is the bridge used by agents to read forum output.

6. **MindSpider as the acquisition subsystem**  
   `MindSpider/` is a separate crawling pipeline for social/media data acquisition. It has two major paths: `BroadTopicExtraction/` for broad topic and news extraction, and `DeepSentimentCrawling/` for deeper platform crawling with keyword management and platform-specific crawlers. Its `schema/` directory defines database structure and ORM models, so this subsystem is both a crawler and the persistence layer that feeds downstream analysis.

7. **Sentiment analysis as a pluggable model family**  
   `SentimentAnalysisModel/` is not a single service but a collection of offline or semi-offline sentiment/topic models: fine-tuned BERT/GPT-2 variants, a multilingual sentiment model, a small-Qwen-based model, and classic ML baselines. In the architecture, these models act as optional analytical backends, especially for `InsightEngine`, rather than as part of the web orchestration path.

8. **ReportEngine as the final assembly and rendering layer**  
   `ReportEngine/` is the last stage of the pipeline. It consumes outputs from the analysis engines and turns them into a structured report through a multi-step process: template selection, document layout, word budgeting, chapter generation, validation, stitching, and rendering. Its `core/` package manages templates and chapter storage; `ir/` defines the intermediate representation contract; `renderers/` produce HTML and PDF, including chart and math conversion utilities.

9. **Intermediate representation and validation boundaries**  
   A key architectural boundary is the ReportEngine IR. Analysis engines emit content that is normalized into structured chapter/document data; the validator layer checks that structure before rendering. This keeps generation, layout, and export decoupled from raw LLM output, and explains the presence of chart/table validators, JSON parsing helpers, and sanitization tests.

10. **External systems and infrastructure dependencies**  
   The system relies on an OpenAI-compatible LLM API across engines, configured via environment variables. Persistence is backed by PostgreSQL by default, with SQLAlchemy-based access in places that need structured querying. Crawling depends on Playwright/Chromium. PDF export depends on WeasyPrint and related system libraries. Docker Compose is the main deployment boundary, packaging the Flask app, database, and supporting services.

11. **Data flow from source data to final report**  
   The typical flow is: crawl or query external/public/private sources → analyze in the three specialized engines → coordinate and refine through the forum layer → collect all outputs into ReportEngine → validate and stitch an IR → render HTML and optionally PDF → store results in `final_reports/` and related artifact directories. The checked-in example report directories show that outputs are treated as reproducible artifacts, not just ephemeral responses.

12. **Operational artifacts and observability**  
   The repository keeps runtime outputs in `logs/`, per-engine report folders, and `final_reports/`. Tests cover selected boundaries such as forum monitoring and report sanitization. This suggests the system’s main reliability concerns are agent coordination integrity and safe report rendering, not just model accuracy.

Main entry points: [`InsightEngine/agent.py`](https://github.com/666ghj/bettafish/blob/main/InsightEngine/agent.py) (Insight engine), [`QueryEngine/agent.py`](https://github.com/666ghj/bettafish/blob/main/QueryEngine/agent.py) (Query engine), [`ReportEngine/agent.py`](https://github.com/666ghj/bettafish/blob/main/ReportEngine/agent.py) (Report engine), [`MediaEngine/agent.py`](https://github.com/666ghj/bettafish/blob/main/MediaEngine/agent.py) (Media engine)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_entry["Entry & ops"]
  node_app_py["Main app<br/>Flask orchestrator<br/>[app.py]"]
  node_single_engine_app["Single engine<br/>Streamlit entry"]
  node_report_only["Report only<br/>rerun entry"]
end

subgraph group_analysis["Analysis engines"]
  node_query_engine["Query engine<br/>web/news analysis<br/>[agent.py]"]
  node_media_engine["Media engine<br/>multimodal analysis<br/>[agent.py]"]
  node_insight_engine["Insight engine<br/>structured mining<br/>[agent.py]"]
  node_qe_search{{"Search steps<br/>engine node<br/>[search_node.py]"}}
  node_me_search{{"Search steps<br/>engine node<br/>[search_node.py]"}}
  node_ie_search{{"Search steps<br/>engine node<br/>[search_node.py]"}}
end

subgraph group_coord["Coordination"]
  node_forum_engine["Forum engine<br/>cross-agent host<br/>[llm_host.py]"]
  node_forum_monitor["Monitor<br/>activity watcher<br/>[monitor.py]"]
end

subgraph group_acq["Acquisition"]
  node_mindspider["MindSpider<br/>crawler pipeline<br/>[main.py]"]
  node_broad_extraction["Broad topics<br/>news extractor<br/>[main.py]"]
  node_deep_crawling["Deep crawl<br/>platform crawler<br/>[main.py]"]
  node_mindspider_schema[("Schema<br/>db models<br/>[models_sa.py]")]
end

subgraph group_report["Reporting"]
  node_report_engine["Report engine<br/>document assembly<br/>[agent.py]"]
  node_template_node{{"Template select<br/>layout node"}}
  node_layout_node{{"Layout"}}
  node_chapter_node{{"Chapters<br/>generation node"}}
  node_ir_layer["IR & validate<br/>report contract<br/>[schema.py]"]
  node_renderer["Renderers<br/>html/pdf export<br/>[html_renderer.py]"]
end

subgraph group_models["Models"]
  node_sentiment_models[("Sentiment models<br/>model family<br/>[predict.py]")]
end

node_app_py -->|"launches"| node_query_engine
node_app_py -->|"launches"| node_media_engine
node_app_py -->|"launches"| node_insight_engine
node_app_py -->|"assembles"| node_report_engine
node_single_engine_app -->|"runs"| node_query_engine
node_single_engine_app -->|"runs"| node_media_engine
node_single_engine_app -->|"runs"| node_insight_engine
node_report_only -->|"rebuilds"| node_report_engine
node_mindspider -->|"collects"| node_broad_extraction
node_mindspider -->|"collects"| node_deep_crawling
node_mindspider -->|"persists"| node_mindspider_schema
node_broad_extraction -->|"feeds"| node_query_engine
node_deep_crawling -->|"feeds"| node_insight_engine
node_forum_monitor -->|"observes"| node_forum_engine
node_forum_engine -->|"coordinates"| node_query_engine
node_forum_engine -->|"coordinates"| node_media_engine
node_forum_engine -->|"coordinates"| node_insight_engine
node_query_engine -->|"executes"| node_qe_search
node_media_engine -->|"executes"| node_me_search
node_insight_engine -->|"executes"| node_ie_search
node_query_engine -->|"outputs"| node_report_engine
node_media_engine -->|"outputs"| node_report_engine
node_insight_engine -->|"outputs"| node_report_engine
node_sentiment_models -->|"supports"| node_insight_engine
node_report_engine -->|"selects"| node_template_node
node_template_node -->|"plans"| node_layout_node
node_layout_node -->|"drives"| node_chapter_node
node_chapter_node -->|"normalizes"| node_ir_layer
node_ir_layer -->|"renders"| node_renderer
node_renderer -->|"produces"| node_report_only

click node_app_py "https://github.com/666ghj/bettafish/blob/main/app.py"
click node_single_engine_app "https://github.com/666ghj/bettafish/blob/main/SingleEngineApp/insight_engine_streamlit_app.py"
click node_report_only "https://github.com/666ghj/bettafish/blob/main/report_engine_only.py"
click node_query_engine "https://github.com/666ghj/bettafish/blob/main/QueryEngine/agent.py"
click node_media_engine "https://github.com/666ghj/bettafish/blob/main/MediaEngine/agent.py"
click node_insight_engine "https://github.com/666ghj/bettafish/blob/main/InsightEngine/agent.py"
click node_qe_search "https://github.com/666ghj/bettafish/blob/main/QueryEngine/nodes/search_node.py"
click node_me_search "https://github.com/666ghj/bettafish/blob/main/MediaEngine/nodes/search_node.py"
click node_ie_search "https://github.com/666ghj/bettafish/blob/main/InsightEngine/nodes/search_node.py"
click node_forum_engine "https://github.com/666ghj/bettafish/blob/main/ForumEngine/llm_host.py"
click node_forum_monitor "https://github.com/666ghj/bettafish/blob/main/ForumEngine/monitor.py"
click node_mindspider "https://github.com/666ghj/bettafish/blob/main/MindSpider/main.py"
click node_broad_extraction "https://github.com/666ghj/bettafish/blob/main/MindSpider/BroadTopicExtraction/main.py"
click node_deep_crawling "https://github.com/666ghj/bettafish/blob/main/MindSpider/DeepSentimentCrawling/main.py"
click node_mindspider_schema "https://github.com/666ghj/bettafish/blob/main/MindSpider/schema/models_sa.py"
click node_report_engine "https://github.com/666ghj/bettafish/blob/main/ReportEngine/agent.py"
click node_template_node "https://github.com/666ghj/bettafish/blob/main/ReportEngine/nodes/template_selection_node.py"
click node_layout_node "https://github.com/666ghj/bettafish/blob/main/ReportEngine/nodes/document_layout_node.py"
click node_chapter_node "https://github.com/666ghj/bettafish/blob/main/ReportEngine/nodes/chapter_generation_node.py"
click node_ir_layer "https://github.com/666ghj/bettafish/blob/main/ReportEngine/ir/schema.py"
click node_renderer "https://github.com/666ghj/bettafish/blob/main/ReportEngine/renderers/html_renderer.py"
click node_sentiment_models "https://github.com/666ghj/bettafish/blob/main/SentimentAnalysisModel/WeiboSentiment_MachineLearning/predict.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_app_py,node_single_engine_app,node_report_only toneBlue
class node_query_engine,node_media_engine,node_insight_engine,node_qe_search,node_me_search,node_ie_search toneAmber
class node_forum_engine,node_forum_monitor toneMint
class node_mindspider,node_broad_extraction,node_deep_crawling,node_mindspider_schema toneRose
class node_report_engine,node_template_node,node_layout_node,node_chapter_node,node_ir_layer,node_renderer toneIndigo
class node_sentiment_models toneTeal
```

## Components

### Entry & ops

Primary UI and rerun entry points for the repository.

- [Main app](https://github.com/666ghj/bettafish/blob/main/app.py): Flask orchestrator · Top-level web entry point that drives the full pipeline. · `app.py`
- [Single engine](https://github.com/666ghj/bettafish/blob/main/SingleEngineApp/insight_engine_streamlit_app.py): Streamlit entry · Standalone Streamlit launchers for running each analysis engine independently. · `SingleEngineApp/insight_engine_streamlit_app.py`
- [Report only](https://github.com/666ghj/bettafish/blob/main/report_engine_only.py): rerun entry · Bypasses analysis and rebuilds outputs from saved artifacts. · `report_engine_only.py`

### Analysis engines

Three parallel agent pipelines that collect and interpret evidence.

- [Query engine](https://github.com/666ghj/bettafish/blob/main/QueryEngine/agent.py): web/news analysis · Broad web and news search plus topic discovery pipeline. · `QueryEngine/agent.py`
- [Media engine](https://github.com/666ghj/bettafish/blob/main/MediaEngine/agent.py): multimodal analysis · Analyzes social, video, and image-oriented public content. · `MediaEngine/agent.py`
- [Insight engine](https://github.com/666ghj/bettafish/blob/main/InsightEngine/agent.py): structured mining · Performs database mining, search over stored records, and deeper structured analysis. · `InsightEngine/agent.py`
- [Search steps](https://github.com/666ghj/bettafish/blob/main/QueryEngine/nodes/search_node.py): engine node · Representative search and evidence-gathering nodes used by the query pipeline. · `QueryEngine/nodes/search_node.py`
- [Search steps](https://github.com/666ghj/bettafish/blob/main/MediaEngine/nodes/search_node.py): engine node · Representative search and evidence-gathering nodes used by the media pipeline. · `MediaEngine/nodes/search_node.py`
- [Search steps](https://github.com/666ghj/bettafish/blob/main/InsightEngine/nodes/search_node.py): engine node · Representative search and evidence-gathering nodes used by the insight pipeline. · `InsightEngine/nodes/search_node.py`

### Coordination

Forum-style cross-agent discussion boundary.

- [Forum engine](https://github.com/666ghj/bettafish/blob/main/ForumEngine/llm_host.py): cross-agent host · Moderated discussion layer that monitors agent activity and coordinates reflection cycles. · `ForumEngine/llm_host.py`
- [Monitor](https://github.com/666ghj/bettafish/blob/main/ForumEngine/monitor.py): activity watcher · Observes agent output and feeds the forum coordination loop. · `ForumEngine/monitor.py`

### Acquisition

Crawling and database-backed data collection pipelines.

- [MindSpider](https://github.com/666ghj/bettafish/blob/main/MindSpider/main.py): crawler pipeline · Acquisition subsystem for broad topic extraction and deep platform crawling. · `MindSpider/main.py`
- [Broad topics](https://github.com/666ghj/bettafish/blob/main/MindSpider/BroadTopicExtraction/main.py): news extractor · Topic and news extraction path within the crawler subsystem. · `MindSpider/BroadTopicExtraction/main.py`
- [Deep crawl](https://github.com/666ghj/bettafish/blob/main/MindSpider/DeepSentimentCrawling/main.py): platform crawler · Platform-specific crawling and keyword management path. · `MindSpider/DeepSentimentCrawling/main.py`
- [Schema](https://github.com/666ghj/bettafish/blob/main/MindSpider/schema/models_sa.py): db models · Database structure and ORM models that persist crawler output. · `MindSpider/schema/models_sa.py`

### Reporting

Intermediate representation, validation, and final rendering.

- [Report engine](https://github.com/666ghj/bettafish/blob/main/ReportEngine/agent.py): document assembly · Transforms analysis outputs into final report documents. · `ReportEngine/agent.py`
- [Template select](https://github.com/666ghj/bettafish/blob/main/ReportEngine/nodes/template_selection_node.py): layout node · Chooses report template and structure. · `ReportEngine/nodes/template_selection_node.py`
- [Layout](https://github.com/666ghj/bettafish/blob/main/ReportEngine/nodes/document_layout_node.py): layout node · Builds document layout and chapter ordering. · `ReportEngine/nodes/document_layout_node.py`
- [Chapters](https://github.com/666ghj/bettafish/blob/main/ReportEngine/nodes/chapter_generation_node.py): generation node · Generates chapter content from the normalized report plan. · `ReportEngine/nodes/chapter_generation_node.py`
- [IR & validate](https://github.com/666ghj/bettafish/blob/main/ReportEngine/ir/schema.py): report contract · Structured intermediate representation and validation boundary before rendering. · `ReportEngine/ir/schema.py`
- [Renderers](https://github.com/666ghj/bettafish/blob/main/ReportEngine/renderers/html_renderer.py): html/pdf export · Produces HTML and PDF output from validated report content. · `ReportEngine/renderers/html_renderer.py`

### Models

Offline and semi-offline sentiment/topic model family.

- [Sentiment models](https://github.com/666ghj/bettafish/blob/main/SentimentAnalysisModel/WeiboSentiment_MachineLearning/predict.py): model family · Optional offline and semi-offline sentiment/topic model backends used by analysis engines. · `SentimentAnalysisModel/WeiboSentiment_MachineLearning/predict.py`

## Connections

- Main app → Query engine: launches · Main app can run the query analysis path.
- Main app → Media engine: launches · Main app can run the media analysis path.
- Main app → Insight engine: launches · Main app can run the insight analysis path.
- Main app → Report engine: assembles · Main app can hand off to report generation.
- Single engine → Query engine: runs · Standalone app for the query engine.
- Single engine → Media engine: runs · Standalone app for the media engine.
- Single engine → Insight engine: runs · Standalone app for the insight engine.
- Report only → Report engine: rebuilds · Report-only workflow regenerates documents from saved artifacts.
- MindSpider → Broad topics: collects · Crawling entry can route to broad topic extraction.
- MindSpider → Deep crawl: collects · Crawling entry can route to deep crawling.
- MindSpider → Schema: persists · Crawler output is stored via schema and ORM models.
- Broad topics → Query engine: feeds · Broad public-source collection feeds the query analysis stream.
- Deep crawl → Insight engine: feeds · Deep crawling and database-backed collection feed insight analysis.
- Monitor → Forum engine: observes · Monitor captures agent activity for the forum host.
- Forum engine → Query engine: coordinates · Forum discussions influence query agent refinement cycles.
- Forum engine → Media engine: coordinates · Forum discussions influence media agent refinement cycles.
- Forum engine → Insight engine: coordinates · Forum discussions influence insight agent refinement cycles.
- Query engine → Search steps: executes · Query engine search and reasoning flow.
- Media engine → Search steps: executes · Media engine search and reasoning flow.
- Insight engine → Search steps: executes · Insight engine search and reasoning flow.
- Query engine → Report engine: outputs · Query results are consumed by the reporting pipeline.
- Media engine → Report engine: outputs · Media results are consumed by the reporting pipeline.
- Insight engine → Report engine: outputs · Insight results are consumed by the reporting pipeline.
- Sentiment models → Insight engine: supports · Optional sentiment and topic models back structured analysis.
- Report engine → Template select: selects · Report assembly begins with template choice.
- Template select → Layout: plans · Template selection informs document layout.
- Layout → Chapters: drives · Layout plan guides chapter generation.
- Chapters → IR & validate: normalizes · Generated content is converted into structured IR.
- IR & validate → Renderers: renders · Validated IR is exported to HTML/PDF renderers.
- Renderers → Report only: produces · Rendered outputs are used by report-only regeneration workflows.
