How rte-france/bertrend works
BERTrend is a platform for monitoring evolving text corpora: it collects and prepares documents, models topics over time, and identifies noise, weak signals, and strong signals for exploration. Its broader product includes prospective-analysis dashboards, topic exploration, newsletters and reports, plus API and background-job services. The README documents the main capabilities; source excerpts and the SOURCE INDEX support some internal links. Several subsystems are unsampled, so their implementation-level connections are intentionally not inferred.
Main entry points: app.py Prospective Dashboard · BERTopicModel.py Topic Modeling · data_provider_utils.py Provider Dispatch · embedding_service.py Embeddings
Components 34
Collection & Preparation
- Feed Providers
bertrend/bertrend_apps/data_provider - Provider Dispatch
bertrend/bertrend_apps/data_provider/data_provider_utils.py - Data Preparation
bertrend/utils/data_loading.py
Topic & Trend Analysis
- Embeddings
bertrend/services/embedding_service.py - Topic Modeling
bertrend/BERTopicModel.py - Temporal Modeling
bertrend/BERTrend.py - Weak Signals
bertrend/trend_analysis/weak_signals.py - Trend Visualizations
bertrend/trend_analysis/visualizations.py - Temporal Metrics
bertrend/metrics/temporal_metrics_embedding.py - Trend Data Structures
bertrend/trend_analysis/data_structure.py
Exploration & Reporting
- Topic Exploration
bertrend/topic_analysis/visualizations.py - Prospective Dashboard
bertrend/bertrend_apps/prospective_demo/app.py - Analysis Dashboard
bertrend/bertrend_apps/prospective_demo/dashboard_analysis.py - Comparative Dashboard
bertrend/bertrend_apps/prospective_demo/dashboard_comparative.py - Signal Dashboard
bertrend/bertrend_apps/prospective_demo/dashboard_signals.py - Report Generation
bertrend/bertrend_apps/prospective_demo/report_generation.py - Newsletter Generation
bertrend/bertrend_apps/newsletters/newsletter_generation.py - New Data Processing
bertrend/bertrend_apps/prospective_demo/process_new_data.py - Curebot Utilities
bertrend/bertrend_apps/exploration/curebot/app_utils.py
Services & Jobs
- Embedding Client
bertrend/services/embedding_client.py - Embedding API
bertrend/services/embedding_server/main.py - BERTrend API
bertrend/bertrend_apps/services/bertrend/bertrend_service.py - Job Queue
bertrend/bertrend_apps/services/queue_management/queue_manager.py - Queue Worker
bertrend/bertrend_apps/services/queue_management/bertrend_worker.py - Queue Monitor
bertrend/bertrend_apps/services/queue_monitor/app.py - Queue Configuration
bertrend/bertrend_apps/services/queue_management/rabbitmq_config.py - Embedding Security
bertrend/services/embedding_server/security.py
Shared Capabilities
- LLM Client
bertrend/llm_utils/openai_client.py - Article Scoring
bertrend/article_scoring/article_scoring.py - Model Parameters
bertrend/config/parameters.py - Newsletter Features
bertrend/llm_utils/newsletter_features.py - Mail Delivery
bertrend/bertrend_apps/common/mail_utils.py
Other components
- Analyst
- Feed Sources
Connections 32
- Analyst → Prospective Dashboard: explores
README.md - Feed Sources → Feed Providers: supply documents
README.md - Feed Providers → Provider Dispatch: dispatches providers
bertrend/bertrend_apps/data_provider/data_provider_utils.py - Provider Dispatch → Article Scoring: uses scoring
bertrend/bertrend_apps/data_provider/data_provider_utils.py - Provider Dispatch → Data Preparation: prepares collected data
- Data Preparation → Embeddings: provides text
- Embeddings → Embedding Client: uses remote client
bertrend/services/embedding_service.py - Embedding Client → Embedding API: requests embeddings
bertrend/services/embedding_client.py - Embedding API → Embedding Security: checks access
- Embeddings → Topic Modeling: embeds documents
- Topic Modeling → Temporal Modeling: models topic evolution
README.md - Temporal Modeling → Weak Signals: classifies signals
README.md - Weak Signals → Trend Data Structures: produces signal data
bertrend/trend_analysis/weak_signals.py - Temporal Modeling → Trend Visualizations: supports trend views
README.md - Topic Modeling → Topic Exploration: supports topic exploration
README.md - Topic Modeling → Temporal Metrics: evaluates stability
README.md - Analyst → Topic Exploration: explores topics
README.md - Prospective Dashboard → Analysis Dashboard: routes to analysis
bertrend/bertrend_apps/prospective_demo/app.py - Prospective Dashboard → Comparative Dashboard: routes to comparison
bertrend/bertrend_apps/prospective_demo/app.py - Prospective Dashboard → Signal Dashboard: routes to signals
bertrend/bertrend_apps/prospective_demo/app.py - Prospective Dashboard → Report Generation: routes to reports
bertrend/bertrend_apps/prospective_demo/app.py - Prospective Dashboard → New Data Processing: monitors new data
- Analyst → Newsletter Generation: requests newsletters
README.md - Newsletter Generation → Newsletter Features: uses features
bertrend/bertrend_apps/newsletters/newsletter_generation.py - Newsletter Generation → Mail Delivery: sends newsletters
bertrend/bertrend_apps/newsletters/newsletter_generation.py - Report Generation → LLM Client: generates reports
- BERTrend API → Job Queue: dispatches jobs
- Job Queue → Queue Configuration: uses configuration
bertrend/bertrend_apps/services/queue_management/queue_manager.py - Queue Worker → Job Queue: consumes jobs
bertrend/bertrend_apps/services/queue_management/bertrend_worker.py - Queue Monitor → Queue Configuration: uses queue settings
bertrend/bertrend_apps/services/queue_monitor/app.py - Topic Modeling → Model Parameters: uses parameters
bertrend/BERTopicModel.py - Curebot Utilities → Newsletter Features: uses newsletter features
bertrend/bertrend_apps/exploration/curebot/tabs/tab3.py