# rte-france/bertrend architecture

> Architecture diagram of the GitHub repository rte-france/bertrend, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-07). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/rte-france/bertrend
- Repository: https://github.com/rte-france/bertrend

## Overview

BERTrend is a platform for monitoring evolving text corpora: it collects and prepares documents, models topics over time, and identifies noise, weak signals, and strong signals for exploration. Its broader product includes prospective-analysis dashboards, topic exploration, newsletters and reports, plus API and background-job services. The README documents the main capabilities; source excerpts and the SOURCE INDEX support some internal links. Several subsystems are unsampled, so their implementation-level connections are intentionally not inferred.

Main entry points: [`bertrend/bertrend_apps/prospective_demo/app.py`](https://github.com/rte-france/bertrend/blob/main/bertrend/bertrend_apps/prospective_demo/app.py) (Prospective Dashboard), [`bertrend/BERTopicModel.py`](https://github.com/rte-france/bertrend/blob/main/bertrend/BERTopicModel.py) (Topic Modeling), [`bertrend/bertrend_apps/data_provider/data_provider_utils.py`](https://github.com/rte-france/bertrend/blob/main/bertrend/bertrend_apps/data_provider/data_provider_utils.py) (Provider Dispatch), [`bertrend/services/embedding_service.py`](https://github.com/rte-france/bertrend/blob/main/bertrend/services/embedding_service.py) (Embeddings)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_collection["Collection &amp; Preparation"]
  node_providers["Feed Providers"]
  node_provider_utils["Provider Dispatch"]
  node_data_prep["Data Preparation<br/>[data_loading.py]"]
end

subgraph group_analysis["Topic &amp; Trend Analysis"]
  node_embedding["Embeddings"]
  node_topic_model["Topic Modeling<br/>[BERTopicModel.py]"]
  node_trend_model["Temporal Modeling<br/>[BERTrend.py]"]
  node_signal_analysis["Weak Signals<br/>[weak_signals.py]"]
  node_trend_visuals["Trend Visualizations<br/>[visualizations.py]"]
  node_metrics["Temporal Metrics"]
  node_trend_data["Trend Data Structures<br/>[data_structure.py]"]
end

subgraph group_exploration["Exploration &amp; Reporting"]
  node_topic_visuals["Topic Exploration<br/>[visualizations.py]"]
  node_prospective_app["Prospective Dashboard<br/>[app.py]"]
  node_dashboard_analysis["Analysis Dashboard"]
  node_dashboard_comparative["Comparative Dashboard"]
  node_dashboard_signals["Signal Dashboard"]
  node_reporting["Report Generation"]
  node_newsletter["Newsletter Generation"]
  node_prospective_processing["New Data Processing"]
  node_demo_utils["Curebot Utilities<br/>[app_utils.py]"]
end

subgraph group_services["Services &amp; Jobs"]
  node_embed_client["Embedding Client"]
  node_embed_api["Embedding API<br/>[main.py]"]
  node_api["BERTrend API"]
  node_queue["Job Queue<br/>[queue_manager.py]"]
  node_worker["Queue Worker<br/>[bertrend_worker.py]"]
  node_queue_monitor["Queue Monitor<br/>[app.py]"]
  node_queue_config["Queue Configuration<br/>[rabbitmq_config.py]"]
  node_security["Embedding Security<br/>[security.py]"]
end

subgraph group_shared["Shared Capabilities"]
  node_llm["LLM Client<br/>[openai_client.py]"]
  node_article_scoring["Article Scoring<br/>[article_scoring.py]"]
  node_parameters["Model Parameters<br/>[parameters.py]"]
  node_newsletter_features["Newsletter Features"]
  node_mail["Mail Delivery<br/>[mail_utils.py]"]
end

node_analyst(("Analyst"))
node_feeds(("Feed Sources"))

node_analyst -->|"explores"| node_prospective_app
node_feeds -->|"supply documents"| node_providers
node_providers -->|"dispatches providers"| node_provider_utils
node_provider_utils -->|"uses scoring"| node_article_scoring
node_provider_utils -.->|"prepares collected data"| node_data_prep
node_data_prep -.->|"provides text"| node_embedding
node_embedding -->|"uses remote client"| node_embed_client
node_embed_client -->|"requests embeddings"| node_embed_api
node_embed_api -->|"checks access"| node_security
node_embedding -.->|"embeds documents"| node_topic_model
node_topic_model -->|"models topic evolution"| node_trend_model
node_trend_model -->|"classifies signals"| node_signal_analysis
node_signal_analysis -->|"produces signal data"| node_trend_data
node_trend_model -->|"supports trend views"| node_trend_visuals
node_topic_model -->|"supports topic exploration"| node_topic_visuals
node_topic_model -->|"evaluates stability"| node_metrics
node_analyst -->|"explores topics"| node_topic_visuals
node_prospective_app -->|"routes to analysis"| node_dashboard_analysis
node_prospective_app -->|"routes to comparison"| node_dashboard_comparative
node_prospective_app -->|"routes to signals"| node_dashboard_signals
node_prospective_app -->|"routes to reports"| node_reporting
node_prospective_app -.->|"monitors new data"| node_prospective_processing
node_analyst -->|"requests newsletters"| node_newsletter
node_newsletter -->|"uses features"| node_newsletter_features
node_newsletter -->|"sends newsletters"| node_mail
node_reporting -.->|"generates reports"| node_llm
node_api -.->|"dispatches jobs"| node_queue
node_queue -->|"uses configuration"| node_queue_config
node_worker -->|"consumes jobs"| node_queue
node_queue_monitor -->|"uses queue settings"| node_queue_config
node_topic_model -->|"uses parameters"| node_parameters
node_demo_utils -->|"uses newsletter features"| node_newsletter_features

click node_providers "https://github.com/rte-france/bertrend/tree/main/bertrend/bertrend_apps/data_provider"
click node_provider_utils "https://github.com/rte-france/bertrend/blob/main/bertrend/bertrend_apps/data_provider/data_provider_utils.py"
click node_data_prep "https://github.com/rte-france/bertrend/blob/main/bertrend/utils/data_loading.py"
click node_embedding "https://github.com/rte-france/bertrend/blob/main/bertrend/services/embedding_service.py"
click node_embed_client "https://github.com/rte-france/bertrend/blob/main/bertrend/services/embedding_client.py"
click node_embed_api "https://github.com/rte-france/bertrend/blob/main/bertrend/services/embedding_server/main.py"
click node_topic_model "https://github.com/rte-france/bertrend/blob/main/bertrend/BERTopicModel.py"
click node_trend_model "https://github.com/rte-france/bertrend/blob/main/bertrend/BERTrend.py"
click node_signal_analysis "https://github.com/rte-france/bertrend/blob/main/bertrend/trend_analysis/weak_signals.py"
click node_trend_visuals "https://github.com/rte-france/bertrend/blob/main/bertrend/trend_analysis/visualizations.py"
click node_topic_visuals "https://github.com/rte-france/bertrend/blob/main/bertrend/topic_analysis/visualizations.py"
click node_metrics "https://github.com/rte-france/bertrend/blob/main/bertrend/metrics/temporal_metrics_embedding.py"
click node_prospective_app "https://github.com/rte-france/bertrend/blob/main/bertrend/bertrend_apps/prospective_demo/app.py"
click node_dashboard_analysis "https://github.com/rte-france/bertrend/blob/main/bertrend/bertrend_apps/prospective_demo/dashboard_analysis.py"
click node_dashboard_comparative "https://github.com/rte-france/bertrend/blob/main/bertrend/bertrend_apps/prospective_demo/dashboard_comparative.py"
click node_dashboard_signals "https://github.com/rte-france/bertrend/blob/main/bertrend/bertrend_apps/prospective_demo/dashboard_signals.py"
click node_reporting "https://github.com/rte-france/bertrend/blob/main/bertrend/bertrend_apps/prospective_demo/report_generation.py"
click node_newsletter "https://github.com/rte-france/bertrend/blob/main/bertrend/bertrend_apps/newsletters/newsletter_generation.py"
click node_llm "https://github.com/rte-france/bertrend/blob/main/bertrend/llm_utils/openai_client.py"
click node_article_scoring "https://github.com/rte-france/bertrend/blob/main/bertrend/article_scoring/article_scoring.py"
click node_api "https://github.com/rte-france/bertrend/blob/main/bertrend/bertrend_apps/services/bertrend/bertrend_service.py"
click node_queue "https://github.com/rte-france/bertrend/blob/main/bertrend/bertrend_apps/services/queue_management/queue_manager.py"
click node_worker "https://github.com/rte-france/bertrend/blob/main/bertrend/bertrend_apps/services/queue_management/bertrend_worker.py"
click node_queue_monitor "https://github.com/rte-france/bertrend/blob/main/bertrend/bertrend_apps/services/queue_monitor/app.py"
click node_queue_config "https://github.com/rte-france/bertrend/blob/main/bertrend/bertrend_apps/services/queue_management/rabbitmq_config.py"
click node_parameters "https://github.com/rte-france/bertrend/blob/main/bertrend/config/parameters.py"
click node_newsletter_features "https://github.com/rte-france/bertrend/blob/main/bertrend/llm_utils/newsletter_features.py"
click node_prospective_processing "https://github.com/rte-france/bertrend/blob/main/bertrend/bertrend_apps/prospective_demo/process_new_data.py"
click node_demo_utils "https://github.com/rte-france/bertrend/blob/main/bertrend/bertrend_apps/exploration/curebot/app_utils.py"
click node_mail "https://github.com/rte-france/bertrend/blob/main/bertrend/bertrend_apps/common/mail_utils.py"
click node_security "https://github.com/rte-france/bertrend/blob/main/bertrend/services/embedding_server/security.py"
click node_trend_data "https://github.com/rte-france/bertrend/blob/main/bertrend/trend_analysis/data_structure.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_providers,node_provider_utils,node_data_prep toneBlue
class node_embedding,node_topic_model,node_trend_model,node_signal_analysis,node_trend_visuals,node_metrics,node_trend_data toneAmber
class node_topic_visuals,node_prospective_app,node_dashboard_analysis,node_dashboard_comparative,node_dashboard_signals,node_reporting,node_newsletter,node_prospective_processing,node_demo_utils toneMint
class node_embed_client,node_embed_api,node_api,node_queue,node_worker,node_queue_monitor,node_queue_config,node_security toneRose
class node_llm,node_article_scoring,node_parameters,node_newsletter_features,node_mail,node_analyst,node_feeds toneIndigo
```

## Components

### Collection & Preparation

- [Feed Providers](https://github.com/rte-france/bertrend/tree/main/bertrend/bertrend_apps/data_provider): `bertrend/bertrend_apps/data_provider`
- [Provider Dispatch](https://github.com/rte-france/bertrend/blob/main/bertrend/bertrend_apps/data_provider/data_provider_utils.py): `bertrend/bertrend_apps/data_provider/data_provider_utils.py`
- [Data Preparation](https://github.com/rte-france/bertrend/blob/main/bertrend/utils/data_loading.py): `bertrend/utils/data_loading.py`

### Topic & Trend Analysis

- [Embeddings](https://github.com/rte-france/bertrend/blob/main/bertrend/services/embedding_service.py): `bertrend/services/embedding_service.py`
- [Topic Modeling](https://github.com/rte-france/bertrend/blob/main/bertrend/BERTopicModel.py): `bertrend/BERTopicModel.py`
- [Temporal Modeling](https://github.com/rte-france/bertrend/blob/main/bertrend/BERTrend.py): `bertrend/BERTrend.py`
- [Weak Signals](https://github.com/rte-france/bertrend/blob/main/bertrend/trend_analysis/weak_signals.py): `bertrend/trend_analysis/weak_signals.py`
- [Trend Visualizations](https://github.com/rte-france/bertrend/blob/main/bertrend/trend_analysis/visualizations.py): `bertrend/trend_analysis/visualizations.py`
- [Temporal Metrics](https://github.com/rte-france/bertrend/blob/main/bertrend/metrics/temporal_metrics_embedding.py): `bertrend/metrics/temporal_metrics_embedding.py`
- [Trend Data Structures](https://github.com/rte-france/bertrend/blob/main/bertrend/trend_analysis/data_structure.py): `bertrend/trend_analysis/data_structure.py`

### Exploration & Reporting

- [Topic Exploration](https://github.com/rte-france/bertrend/blob/main/bertrend/topic_analysis/visualizations.py): `bertrend/topic_analysis/visualizations.py`
- [Prospective Dashboard](https://github.com/rte-france/bertrend/blob/main/bertrend/bertrend_apps/prospective_demo/app.py): `bertrend/bertrend_apps/prospective_demo/app.py`
- [Analysis Dashboard](https://github.com/rte-france/bertrend/blob/main/bertrend/bertrend_apps/prospective_demo/dashboard_analysis.py): `bertrend/bertrend_apps/prospective_demo/dashboard_analysis.py`
- [Comparative Dashboard](https://github.com/rte-france/bertrend/blob/main/bertrend/bertrend_apps/prospective_demo/dashboard_comparative.py): `bertrend/bertrend_apps/prospective_demo/dashboard_comparative.py`
- [Signal Dashboard](https://github.com/rte-france/bertrend/blob/main/bertrend/bertrend_apps/prospective_demo/dashboard_signals.py): `bertrend/bertrend_apps/prospective_demo/dashboard_signals.py`
- [Report Generation](https://github.com/rte-france/bertrend/blob/main/bertrend/bertrend_apps/prospective_demo/report_generation.py): `bertrend/bertrend_apps/prospective_demo/report_generation.py`
- [Newsletter Generation](https://github.com/rte-france/bertrend/blob/main/bertrend/bertrend_apps/newsletters/newsletter_generation.py): `bertrend/bertrend_apps/newsletters/newsletter_generation.py`
- [New Data Processing](https://github.com/rte-france/bertrend/blob/main/bertrend/bertrend_apps/prospective_demo/process_new_data.py): `bertrend/bertrend_apps/prospective_demo/process_new_data.py`
- [Curebot Utilities](https://github.com/rte-france/bertrend/blob/main/bertrend/bertrend_apps/exploration/curebot/app_utils.py): `bertrend/bertrend_apps/exploration/curebot/app_utils.py`

### Services & Jobs

- [Embedding Client](https://github.com/rte-france/bertrend/blob/main/bertrend/services/embedding_client.py): `bertrend/services/embedding_client.py`
- [Embedding API](https://github.com/rte-france/bertrend/blob/main/bertrend/services/embedding_server/main.py): `bertrend/services/embedding_server/main.py`
- [BERTrend API](https://github.com/rte-france/bertrend/blob/main/bertrend/bertrend_apps/services/bertrend/bertrend_service.py): `bertrend/bertrend_apps/services/bertrend/bertrend_service.py`
- [Job Queue](https://github.com/rte-france/bertrend/blob/main/bertrend/bertrend_apps/services/queue_management/queue_manager.py): `bertrend/bertrend_apps/services/queue_management/queue_manager.py`
- [Queue Worker](https://github.com/rte-france/bertrend/blob/main/bertrend/bertrend_apps/services/queue_management/bertrend_worker.py): `bertrend/bertrend_apps/services/queue_management/bertrend_worker.py`
- [Queue Monitor](https://github.com/rte-france/bertrend/blob/main/bertrend/bertrend_apps/services/queue_monitor/app.py): `bertrend/bertrend_apps/services/queue_monitor/app.py`
- [Queue Configuration](https://github.com/rte-france/bertrend/blob/main/bertrend/bertrend_apps/services/queue_management/rabbitmq_config.py): `bertrend/bertrend_apps/services/queue_management/rabbitmq_config.py`
- [Embedding Security](https://github.com/rte-france/bertrend/blob/main/bertrend/services/embedding_server/security.py): `bertrend/services/embedding_server/security.py`

### Shared Capabilities

- [LLM Client](https://github.com/rte-france/bertrend/blob/main/bertrend/llm_utils/openai_client.py): `bertrend/llm_utils/openai_client.py`
- [Article Scoring](https://github.com/rte-france/bertrend/blob/main/bertrend/article_scoring/article_scoring.py): `bertrend/article_scoring/article_scoring.py`
- [Model Parameters](https://github.com/rte-france/bertrend/blob/main/bertrend/config/parameters.py): `bertrend/config/parameters.py`
- [Newsletter Features](https://github.com/rte-france/bertrend/blob/main/bertrend/llm_utils/newsletter_features.py): `bertrend/llm_utils/newsletter_features.py`
- [Mail Delivery](https://github.com/rte-france/bertrend/blob/main/bertrend/bertrend_apps/common/mail_utils.py): `bertrend/bertrend_apps/common/mail_utils.py`

### Other components

- **Analyst**
- **Feed Sources**

## Connections

- Analyst → Prospective Dashboard: explores (evidence: `README.md`)
- Feed Sources → Feed Providers: supply documents (evidence: `README.md`)
- Feed Providers → Provider Dispatch: dispatches providers (evidence: `bertrend/bertrend_apps/data_provider/data_provider_utils.py`)
- Provider Dispatch → Article Scoring: uses scoring (evidence: `bertrend/bertrend_apps/data_provider/data_provider_utils.py`)
- Provider Dispatch → Data Preparation: prepares collected data
- Data Preparation → Embeddings: provides text
- Embeddings → Embedding Client: uses remote client (evidence: `bertrend/services/embedding_service.py`)
- Embedding Client → Embedding API: requests embeddings (evidence: `bertrend/services/embedding_client.py`)
- Embedding API → Embedding Security: checks access
- Embeddings → Topic Modeling: embeds documents
- Topic Modeling → Temporal Modeling: models topic evolution (evidence: `README.md`)
- Temporal Modeling → Weak Signals: classifies signals (evidence: `README.md`)
- Weak Signals → Trend Data Structures: produces signal data (evidence: `bertrend/trend_analysis/weak_signals.py`)
- Temporal Modeling → Trend Visualizations: supports trend views (evidence: `README.md`)
- Topic Modeling → Topic Exploration: supports topic exploration (evidence: `README.md`)
- Topic Modeling → Temporal Metrics: evaluates stability (evidence: `README.md`)
- Analyst → Topic Exploration: explores topics (evidence: `README.md`)
- Prospective Dashboard → Analysis Dashboard: routes to analysis (evidence: `bertrend/bertrend_apps/prospective_demo/app.py`)
- Prospective Dashboard → Comparative Dashboard: routes to comparison (evidence: `bertrend/bertrend_apps/prospective_demo/app.py`)
- Prospective Dashboard → Signal Dashboard: routes to signals (evidence: `bertrend/bertrend_apps/prospective_demo/app.py`)
- Prospective Dashboard → Report Generation: routes to reports (evidence: `bertrend/bertrend_apps/prospective_demo/app.py`)
- Prospective Dashboard → New Data Processing: monitors new data
- Analyst → Newsletter Generation: requests newsletters (evidence: `README.md`)
- Newsletter Generation → Newsletter Features: uses features (evidence: `bertrend/bertrend_apps/newsletters/newsletter_generation.py`)
- Newsletter Generation → Mail Delivery: sends newsletters (evidence: `bertrend/bertrend_apps/newsletters/newsletter_generation.py`)
- Report Generation → LLM Client: generates reports
- BERTrend API → Job Queue: dispatches jobs
- Job Queue → Queue Configuration: uses configuration (evidence: `bertrend/bertrend_apps/services/queue_management/queue_manager.py`)
- Queue Worker → Job Queue: consumes jobs (evidence: `bertrend/bertrend_apps/services/queue_management/bertrend_worker.py`)
- Queue Monitor → Queue Configuration: uses queue settings (evidence: `bertrend/bertrend_apps/services/queue_monitor/app.py`)
- Topic Modeling → Model Parameters: uses parameters (evidence: `bertrend/BERTopicModel.py`)
- Curebot Utilities → Newsletter Features: uses newsletter features (evidence: `bertrend/bertrend_apps/exploration/curebot/tabs/tab3.py`)
