# deshawa77/world-pulse-research architecture

> Architecture diagram of the GitHub repository deshawa77/world-pulse-research, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-22). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/deshawa77/world-pulse-research
- Repository: https://github.com/deshawa77/world-pulse-research

## Overview

World Pulse is a global and country-level crisis-intelligence product: it gathers public, economic, environmental, and market signals, derives risk and forecasts, then exposes results through APIs and an operator dashboard. The principal workflow runs from collectors through streaming/orchestration and feature analysis to persistence and serving; separate branches support country intelligence, scenarios, model governance, and monitoring. The source excerpts and index support several backend relationships, but the README is empty and frontend-to-backend/API details are not shown, so those links are omitted or marked unverified.

Main entry points: [`orchestrator.py`](https://github.com/deshawa77/world-pulse-research/blob/main/orchestrator.py) (Pipeline orchestrator), [`database/mongo.py`](https://github.com/deshawa77/world-pulse-research/blob/main/database/mongo.py) (MongoDB store), [`backend/kafka_client.py`](https://github.com/deshawa77/world-pulse-research/blob/main/backend/kafka_client.py) (Kafka transport), [`backend/main.py`](https://github.com/deshawa77/world-pulse-research/blob/main/backend/main.py) (API and intelligence service)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_ingest["Signal ingestion"]
  node_orchestrator["Pipeline orchestrator<br/>[orchestrator.py]"]
  node_collectors["Signal collectors"]
  node_coingecko["Crypto collector<br/>[coingecko.py]"]
  node_weather["Weather collector<br/>[weather.py]"]
  node_who["Health collector<br/>[who.py]"]
  node_trends["Trend collector<br/>[trends.py]"]
  node_kafka["Kafka transport<br/>[kafka_client.py]"]
end

subgraph group_analysis["Risk intelligence"]
  node_preprocess["Data preprocessing<br/>[preprocess_data.py]"]
  node_topic["Topic intelligence"]
  node_robust_topics["Robust topic pipeline"]
  node_daily_features["Feature builder"]
  node_global_risk["Global risk<br/>[global_risk.py]"]
  node_crisis_detector["Crisis detector"]
  node_country_risk["Country risk"]
  node_causal["Causal explanation"]
  node_counterfactual["Scenario analysis"]
end

subgraph group_serving["Serving and storage"]
  node_backend["API and intelligence service<br/>[main.py]"]
  node_mongo[("MongoDB store<br/>[mongo.py]")]
  node_feature_store[("Feature store<br/>[feature_store.py]")]
end

subgraph group_experience["Operator experience"]
  node_frontend["Web application<br/>[App.tsx]"]
  node_api_client["Frontend API client<br/>[api.ts]"]
  node_dashboard["Intelligence dashboard<br/>[Dashboard.tsx]"]
  node_prediction["Prediction views"]
end

subgraph group_governance["Models and operations"]
  node_model_registry["Model registry<br/>[model_registry.py]"]
  node_scheduler["Scheduled jobs<br/>[runner.py]"]
  node_observability["Pipeline observability<br/>[observability.py]"]
end

node_operator(("Operator"))

node_orchestrator -->|"runs collectors"| node_collectors
node_orchestrator -->|"starts stream"| node_kafka
node_orchestrator -->|"calls preprocessing"| node_preprocess
node_orchestrator -->|"runs modeling"| node_topic
node_orchestrator -->|"builds features"| node_daily_features
node_orchestrator -->|"runs risk analysis"| node_global_risk
node_orchestrator -->|"writes features"| node_feature_store
node_orchestrator -->|"reports events"| node_observability
node_coingecko -->|"publishes signals"| node_kafka
node_coingecko -->|"stores records"| node_mongo
node_weather -->|"publishes signals"| node_kafka
node_weather -->|"stores records"| node_mongo
node_who -->|"publishes signals"| node_kafka
node_who -->|"stores records"| node_mongo
node_trends -->|"publishes signals"| node_kafka
node_trends -->|"stores records"| node_mongo
node_daily_features -->|"reads data"| node_mongo
node_daily_features -->|"uses risk features"| node_global_risk
node_crisis_detector -->|"reads records"| node_mongo
node_counterfactual -->|"gets explanation"| node_causal
node_backend -->|"serves country risk"| node_country_risk
node_backend -->|"serves global risk"| node_global_risk
node_backend -->|"reads and writes"| node_mongo
node_frontend -->|"routes to"| node_dashboard
node_frontend -->|"routes to"| node_prediction
node_dashboard -.->|"uses client"| node_api_client
node_prediction -.->|"uses client"| node_api_client
node_api_client -.->|"requests data"| node_backend
node_scheduler -->|"runs validation"| node_country_risk
node_scheduler -.->|"manages models"| node_model_registry
node_operator -.->|"uses"| node_frontend

click node_orchestrator "https://github.com/deshawa77/world-pulse-research/blob/main/orchestrator.py"
click node_collectors "https://github.com/deshawa77/world-pulse-research/tree/main/collectors"
click node_coingecko "https://github.com/deshawa77/world-pulse-research/blob/main/collectors/coingecko.py"
click node_weather "https://github.com/deshawa77/world-pulse-research/blob/main/collectors/weather.py"
click node_who "https://github.com/deshawa77/world-pulse-research/blob/main/collectors/who.py"
click node_trends "https://github.com/deshawa77/world-pulse-research/blob/main/collectors/trends.py"
click node_kafka "https://github.com/deshawa77/world-pulse-research/blob/main/backend/kafka_client.py"
click node_preprocess "https://github.com/deshawa77/world-pulse-research/blob/main/processing/preprocess_data.py"
click node_topic "https://github.com/deshawa77/world-pulse-research/blob/main/processing/topic_modeling_with_nlp.py"
click node_robust_topics "https://github.com/deshawa77/world-pulse-research/blob/main/processing/robust_topic_pipeline.py"
click node_daily_features "https://github.com/deshawa77/world-pulse-research/blob/main/processing/daily_feature_builder.py"
click node_global_risk "https://github.com/deshawa77/world-pulse-research/blob/main/processing/global_risk.py"
click node_crisis_detector "https://github.com/deshawa77/world-pulse-research/blob/main/processing/global_crisis_detector.py"
click node_country_risk "https://github.com/deshawa77/world-pulse-research/blob/main/processing/country_daily_risk.py"
click node_causal "https://github.com/deshawa77/world-pulse-research/blob/main/processing/causal_risk_navigator.py"
click node_counterfactual "https://github.com/deshawa77/world-pulse-research/blob/main/processing/counterfactual_engine.py"
click node_backend "https://github.com/deshawa77/world-pulse-research/blob/main/backend/main.py"
click node_mongo "https://github.com/deshawa77/world-pulse-research/blob/main/database/mongo.py"
click node_feature_store "https://github.com/deshawa77/world-pulse-research/blob/main/feature_store/feature_store.py"
click node_frontend "https://github.com/deshawa77/world-pulse-research/blob/main/world-pulse-frontend/src/App.tsx"
click node_api_client "https://github.com/deshawa77/world-pulse-research/blob/main/world-pulse-frontend/src/services/api.ts"
click node_dashboard "https://github.com/deshawa77/world-pulse-research/blob/main/world-pulse-frontend/src/pages/Dashboard.tsx"
click node_prediction "https://github.com/deshawa77/world-pulse-research/blob/main/world-pulse-frontend/src/pages/TrendPrediction.tsx"
click node_model_registry "https://github.com/deshawa77/world-pulse-research/blob/main/feature_store/model_registry.py"
click node_scheduler "https://github.com/deshawa77/world-pulse-research/blob/main/scheduler/runner.py"
click node_observability "https://github.com/deshawa77/world-pulse-research/blob/main/backend/observability.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_orchestrator,node_collectors,node_coingecko,node_weather,node_who,node_trends,node_kafka toneBlue
class node_preprocess,node_topic,node_robust_topics,node_daily_features,node_global_risk,node_crisis_detector,node_country_risk,node_causal,node_counterfactual toneAmber
class node_backend,node_mongo,node_feature_store toneMint
class node_frontend,node_api_client,node_dashboard,node_prediction toneRose
class node_model_registry,node_scheduler,node_observability,node_operator toneIndigo
```

## Components

### Signal ingestion

- [Pipeline orchestrator](https://github.com/deshawa77/world-pulse-research/blob/main/orchestrator.py): `orchestrator.py`
- [Signal collectors](https://github.com/deshawa77/world-pulse-research/tree/main/collectors): `collectors`
- [Crypto collector](https://github.com/deshawa77/world-pulse-research/blob/main/collectors/coingecko.py): `collectors/coingecko.py`
- [Weather collector](https://github.com/deshawa77/world-pulse-research/blob/main/collectors/weather.py): `collectors/weather.py`
- [Health collector](https://github.com/deshawa77/world-pulse-research/blob/main/collectors/who.py): `collectors/who.py`
- [Trend collector](https://github.com/deshawa77/world-pulse-research/blob/main/collectors/trends.py): `collectors/trends.py`
- [Kafka transport](https://github.com/deshawa77/world-pulse-research/blob/main/backend/kafka_client.py): `backend/kafka_client.py`

### Risk intelligence

- [Data preprocessing](https://github.com/deshawa77/world-pulse-research/blob/main/processing/preprocess_data.py): `processing/preprocess_data.py`
- [Topic intelligence](https://github.com/deshawa77/world-pulse-research/blob/main/processing/topic_modeling_with_nlp.py): `processing/topic_modeling_with_nlp.py`
- [Robust topic pipeline](https://github.com/deshawa77/world-pulse-research/blob/main/processing/robust_topic_pipeline.py): `processing/robust_topic_pipeline.py`
- [Feature builder](https://github.com/deshawa77/world-pulse-research/blob/main/processing/daily_feature_builder.py): `processing/daily_feature_builder.py`
- [Global risk](https://github.com/deshawa77/world-pulse-research/blob/main/processing/global_risk.py): `processing/global_risk.py`
- [Crisis detector](https://github.com/deshawa77/world-pulse-research/blob/main/processing/global_crisis_detector.py): `processing/global_crisis_detector.py`
- [Country risk](https://github.com/deshawa77/world-pulse-research/blob/main/processing/country_daily_risk.py): `processing/country_daily_risk.py`
- [Causal explanation](https://github.com/deshawa77/world-pulse-research/blob/main/processing/causal_risk_navigator.py): `processing/causal_risk_navigator.py`
- [Scenario analysis](https://github.com/deshawa77/world-pulse-research/blob/main/processing/counterfactual_engine.py): `processing/counterfactual_engine.py`

### Serving and storage

- [API and intelligence service](https://github.com/deshawa77/world-pulse-research/blob/main/backend/main.py): `backend/main.py`
- [MongoDB store](https://github.com/deshawa77/world-pulse-research/blob/main/database/mongo.py): `database/mongo.py`
- [Feature store](https://github.com/deshawa77/world-pulse-research/blob/main/feature_store/feature_store.py): `feature_store/feature_store.py`

### Operator experience

- [Web application](https://github.com/deshawa77/world-pulse-research/blob/main/world-pulse-frontend/src/App.tsx): `world-pulse-frontend/src/App.tsx`
- [Frontend API client](https://github.com/deshawa77/world-pulse-research/blob/main/world-pulse-frontend/src/services/api.ts): `world-pulse-frontend/src/services/api.ts`
- [Intelligence dashboard](https://github.com/deshawa77/world-pulse-research/blob/main/world-pulse-frontend/src/pages/Dashboard.tsx): `world-pulse-frontend/src/pages/Dashboard.tsx`
- [Prediction views](https://github.com/deshawa77/world-pulse-research/blob/main/world-pulse-frontend/src/pages/TrendPrediction.tsx): `world-pulse-frontend/src/pages/TrendPrediction.tsx`

### Models and operations

- [Model registry](https://github.com/deshawa77/world-pulse-research/blob/main/feature_store/model_registry.py): `feature_store/model_registry.py`
- [Scheduled jobs](https://github.com/deshawa77/world-pulse-research/blob/main/scheduler/runner.py): `scheduler/runner.py`
- [Pipeline observability](https://github.com/deshawa77/world-pulse-research/blob/main/backend/observability.py): `backend/observability.py`

### Other components

- **Operator**

## Connections

- Pipeline orchestrator → Signal collectors: runs collectors (evidence: `orchestrator.py`)
- Pipeline orchestrator → Kafka transport: starts stream (evidence: `orchestrator.py`)
- Pipeline orchestrator → Data preprocessing: calls preprocessing (evidence: `orchestrator.py`)
- Pipeline orchestrator → Topic intelligence: runs modeling (evidence: `orchestrator.py`)
- Pipeline orchestrator → Feature builder: builds features (evidence: `orchestrator.py`)
- Pipeline orchestrator → Global risk: runs risk analysis (evidence: `orchestrator.py`)
- Pipeline orchestrator → Feature store: writes features (evidence: `orchestrator.py`)
- Pipeline orchestrator → Pipeline observability: reports events (evidence: `orchestrator.py`)
- Crypto collector → Kafka transport: publishes signals (evidence: `collectors/coingecko.py`)
- Crypto collector → MongoDB store: stores records (evidence: `collectors/coingecko.py`)
- Weather collector → Kafka transport: publishes signals (evidence: `collectors/weather.py`)
- Weather collector → MongoDB store: stores records (evidence: `collectors/weather.py`)
- Health collector → Kafka transport: publishes signals (evidence: `collectors/who.py`)
- Health collector → MongoDB store: stores records (evidence: `collectors/who.py`)
- Trend collector → Kafka transport: publishes signals (evidence: `collectors/trends.py`)
- Trend collector → MongoDB store: stores records (evidence: `collectors/trends.py`)
- Feature builder → MongoDB store: reads data (evidence: `processing/daily_feature_builder.py`)
- Feature builder → Global risk: uses risk features (evidence: `processing/daily_feature_builder.py`)
- Crisis detector → MongoDB store: reads records (evidence: `processing/global_crisis_detector.py`)
- Scenario analysis → Causal explanation: gets explanation (evidence: `processing/counterfactual_engine.py`)
- API and intelligence service → Country risk: serves country risk (evidence: `backend/main.py`)
- API and intelligence service → Global risk: serves global risk (evidence: `backend/main.py`)
- API and intelligence service → MongoDB store: reads and writes (evidence: `backend/main.py`)
- Web application → Intelligence dashboard: routes to (evidence: `world-pulse-frontend/src/App.tsx`)
- Web application → Prediction views: routes to (evidence: `world-pulse-frontend/src/App.tsx`)
- Intelligence dashboard → Frontend API client: uses client
- Prediction views → Frontend API client: uses client
- Frontend API client → API and intelligence service: requests data
- Scheduled jobs → Country risk: runs validation (evidence: `scheduler/runner.py`)
- Scheduled jobs → Model registry: manages models
- Operator → Web application: uses
