How 666ghj/bettafish works
1. Repository purpose and top-level shape
BettaFish is a multi-agent opinion-analysis system. The root app orchestrates a full pipeline from data acquisition and analysis to report generation. The repository is split into four main runtime domains: crawling/data collection, three specialized analysis engines, a forum-style coordination layer, and a report engine that converts analysis outputs into final HTML/PDF deliverables.
2. User-facing entry points
The main web entry is app.py, backed by a Flask UI in templates/ and static/. There are also standalone Streamlit entry points under SingleEngineApp/ for running each analysis engine independently. For report-only workflows, report_engine_only.py and the regenerate_latest_*.py scripts bypass the analysis step and rebuild outputs from saved artifacts.
3. Three analysis engines with a shared architecture pattern
QueryEngine/, MediaEngine/, and InsightEngine/ follow the same internal structure: an agent.py orchestrator, llms/ abstraction, nodes/ for stepwise reasoning, tools/ for external actions, prompts/ for prompt templates, state/ for runtime state, and utils/ for configuration and text handling. This consistent layout makes them separate but parallel subsystems, each specializing in a different data source or analysis style.
4. What each analysis engine is responsible for
QueryEngine focuses on broad web/news search and topic discovery.
MediaEngine focuses on multimodal content analysis, especially social/video/image-oriented public content.
InsightEngine focuses on private database mining and deeper structured analysis, including search over stored records and sentiment analysis integration.
Architecturally, these engines are peer services: they collect different evidence streams, then produce intermediate markdown/state outputs consumed later by reporting.
5. ForumEngine as the coordination boundary
ForumEngine/ is the cross-agent collaboration layer. It monitors agent activity and uses an LLM “host” to simulate a moderated forum discussion. The repository’s core idea is that agents do not only work independently; they also exchange findings through this shared coordination mechanism, which influences subsequent search/reflection cycles. The utils/forum_reader.py helper is the bridge used by agents to read forum output.
6. MindSpider as the acquisition subsystem
MindSpider/ is a separate crawling pipeline for social/media data acquisition. It has two major paths: BroadTopicExtraction/ for broad topic and news extraction, and DeepSentimentCrawling/ for deeper platform crawling with keyword management and platform-specific crawlers. Its schema/ directory defines database structure and ORM models, so this subsystem is both a crawler and the persistence layer that feeds downstream analysis.
7. Sentiment analysis as a pluggable model family
SentimentAnalysisModel/ is not a single service but a collection of offline or semi-offline sentiment/topic models: fine-tuned BERT/GPT-2 variants, a multilingual sentiment model, a small-Qwen-based model, and classic ML baselines. In the architecture, these models act as optional analytical backends, especially for InsightEngine, rather than as part of the web orchestration path.
8. ReportEngine as the final assembly and rendering layer
ReportEngine/ is the last stage of the pipeline. It consumes outputs from the analysis engines and turns them into a structured report through a multi-step process: template selection, document layout, word budgeting, chapter generation, validation, stitching, and rendering. Its core/ package manages templates and chapter storage; ir/ defines the intermediate representation contract; renderers/ produce HTML and PDF, including chart and math conversion utilities.
9. Intermediate representation and validation boundaries
A key architectural boundary is the ReportEngine IR. Analysis engines emit content that is normalized into structured chapter/document data; the validator layer checks that structure before rendering. This keeps generation, layout, and export decoupled from raw LLM output, and explains the presence of chart/table validators, JSON parsing helpers, and sanitization tests.
10. External systems and infrastructure dependencies
The system relies on an OpenAI-compatible LLM API across engines, configured via environment variables. Persistence is backed by PostgreSQL by default, with SQLAlchemy-based access in places that need structured querying. Crawling depends on Playwright/Chromium. PDF export depends on WeasyPrint and related system libraries. Docker Compose is the main deployment boundary, packaging the Flask app, database, and supporting services.
11. Data flow from source data to final report
The typical flow is: crawl or query external/public/private sources → analyze in the three specialized engines → coordinate and refine through the forum layer → collect all outputs into ReportEngine → validate and stitch an IR → render HTML and optionally PDF → store results in final_reports/ and related artifact directories. The checked-in example report directories show that outputs are treated as reproducible artifacts, not just ephemeral responses.
12. Operational artifacts and observability
The repository keeps runtime outputs in logs/, per-engine report folders, and final_reports/. Tests cover selected boundaries such as forum monitoring and report sanitization. This suggests the system’s main reliability concerns are agent coordination integrity and safe report rendering, not just model accuracy.
Main entry points: InsightEngine/agent.py Insight engine · QueryEngine/agent.py Query engine · ReportEngine/agent.py Report engine · MediaEngine/agent.py Media engine
Components 22
Entry & ops
Primary UI and rerun entry points for the repository.
- Main app · Top-level web entry point that drives the full pipeline.
app.py - Single engine · Standalone Streamlit launchers for running each analysis engine independently.
SingleEngineApp/insight_engine_streamlit_app.py - Report only · Bypasses analysis and rebuilds outputs from saved artifacts.
report_engine_only.py
Analysis engines
Three parallel agent pipelines that collect and interpret evidence.
- Query engine · Broad web and news search plus topic discovery pipeline.
QueryEngine/agent.py - Media engine · Analyzes social, video, and image-oriented public content.
MediaEngine/agent.py - Insight engine · Performs database mining, search over stored records, and deeper structured analysis.
InsightEngine/agent.py - Search steps · Representative search and evidence-gathering nodes used by the query pipeline.
QueryEngine/nodes/search_node.py - Search steps · Representative search and evidence-gathering nodes used by the media pipeline.
MediaEngine/nodes/search_node.py - Search steps · Representative search and evidence-gathering nodes used by the insight pipeline.
InsightEngine/nodes/search_node.py
Coordination
Forum-style cross-agent discussion boundary.
- Forum engine · Moderated discussion layer that monitors agent activity and coordinates reflection cycles.
ForumEngine/llm_host.py - Monitor · Observes agent output and feeds the forum coordination loop.
ForumEngine/monitor.py
Acquisition
Crawling and database-backed data collection pipelines.
- MindSpider · Acquisition subsystem for broad topic extraction and deep platform crawling.
MindSpider/main.py - Broad topics · Topic and news extraction path within the crawler subsystem.
MindSpider/BroadTopicExtraction/main.py - Deep crawl · Platform-specific crawling and keyword management path.
MindSpider/DeepSentimentCrawling/main.py - Schema · Database structure and ORM models that persist crawler output.
MindSpider/schema/models_sa.py
Reporting
Intermediate representation, validation, and final rendering.
- Report engine · Transforms analysis outputs into final report documents.
ReportEngine/agent.py - Template select · Chooses report template and structure.
ReportEngine/nodes/template_selection_node.py - Layout · Builds document layout and chapter ordering.
ReportEngine/nodes/document_layout_node.py - Chapters · Generates chapter content from the normalized report plan.
ReportEngine/nodes/chapter_generation_node.py - IR & validate · Structured intermediate representation and validation boundary before rendering.
ReportEngine/ir/schema.py - Renderers · Produces HTML and PDF output from validated report content.
ReportEngine/renderers/html_renderer.py
Models
Offline and semi-offline sentiment/topic model family.
- Sentiment models · Optional offline and semi-offline sentiment/topic model backends used by analysis engines.
SentimentAnalysisModel/WeiboSentiment_MachineLearning/predict.py
Connections 30
- Main app → Query engine: launches · Main app can run the query analysis path.
- Main app → Media engine: launches · Main app can run the media analysis path.
- Main app → Insight engine: launches · Main app can run the insight analysis path.
- Main app → Report engine: assembles · Main app can hand off to report generation.
- Single engine → Query engine: runs · Standalone app for the query engine.
- Single engine → Media engine: runs · Standalone app for the media engine.
- Single engine → Insight engine: runs · Standalone app for the insight engine.
- Report only → Report engine: rebuilds · Report-only workflow regenerates documents from saved artifacts.
- MindSpider → Broad topics: collects · Crawling entry can route to broad topic extraction.
- MindSpider → Deep crawl: collects · Crawling entry can route to deep crawling.
- MindSpider → Schema: persists · Crawler output is stored via schema and ORM models.
- Broad topics → Query engine: feeds · Broad public-source collection feeds the query analysis stream.
- Deep crawl → Insight engine: feeds · Deep crawling and database-backed collection feed insight analysis.
- Monitor → Forum engine: observes · Monitor captures agent activity for the forum host.
- Forum engine → Query engine: coordinates · Forum discussions influence query agent refinement cycles.
- Forum engine → Media engine: coordinates · Forum discussions influence media agent refinement cycles.
- Forum engine → Insight engine: coordinates · Forum discussions influence insight agent refinement cycles.
- Query engine → Search steps: executes · Query engine search and reasoning flow.
- Media engine → Search steps: executes · Media engine search and reasoning flow.
- Insight engine → Search steps: executes · Insight engine search and reasoning flow.
- Query engine → Report engine: outputs · Query results are consumed by the reporting pipeline.
- Media engine → Report engine: outputs · Media results are consumed by the reporting pipeline.
- Insight engine → Report engine: outputs · Insight results are consumed by the reporting pipeline.
- Sentiment models → Insight engine: supports · Optional sentiment and topic models back structured analysis.
- Report engine → Template select: selects · Report assembly begins with template choice.
- Template select → Layout: plans · Template selection informs document layout.
- Layout → Chapters: drives · Layout plan guides chapter generation.
- Chapters → IR & validate: normalizes · Generated content is converted into structured IR.
- IR & validate → Renderers: renders · Validated IR is exported to HTML/PDF renderers.
- Renderers → Report only: produces · Rendered outputs are used by report-only regeneration workflows.