# charankumarkharade/dax40-sentiment-monitor architecture

> Architecture diagram of the GitHub repository charankumarkharade/dax40-sentiment-monitor, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/charankumarkharade/dax40-sentiment-monitor
- Repository: https://github.com/charankumarkharade/dax40-sentiment-monitor

## Overview

This repository describes a DAX 40 news-monitoring pipeline: it collects Tagesschau articles, cleans and enriches them with sentiment and company attribution, models results for analytics, and supports alerts and BI. Notebook and SQL contents were not sampled, so their internal wiring is shown only where the README documents it; SQL capabilities are included without inferred data-flow links.

Main entry points: [`notebooks/Polished_Gold.ipynb`](https://github.com/charankumarkharade/dax40-sentiment-monitor/blob/main/notebooks/Polished_Gold.ipynb) (Gold enrichment), [`notebooks/Cleaning_Silver.ipynb`](https://github.com/charankumarkharade/dax40-sentiment-monitor/blob/main/notebooks/Cleaning_Silver.ipynb) (Silver cleaning), [`notebooks/Star_models.ipynb`](https://github.com/charankumarkharade/dax40-sentiment-monitor/blob/main/notebooks/Star_models.ipynb) (Star modeling), [`notebooks/Ingestion_Bronze.ipynb`](https://github.com/charankumarkharade/dax40-sentiment-monitor/blob/main/notebooks/Ingestion_Bronze.ipynb) (Bronze ingestion)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_ingest["Ingestion and orchestration"]
  node_pipeline["Fabric pipeline"]
  node_bronze_nb["Bronze ingestion"]
end

subgraph group_data["Medallion data stages"]
  node_bronze_store[("Bronze archive")]
  node_silver_nb["Silver cleaning"]
  node_silver_store[("Silver articles")]
  node_gold_nb["Gold enrichment"]
  node_company_aliases["Company aliases<br/>[dax40_companies.py]"]
  node_gold_store[("Gold analytics")]
end

subgraph group_analytics["Analytics and risk"]
  node_star_nb["Star modeling<br/>[Star_models.ipynb]"]
  node_star_store[("Star schema")]
  node_reputation_sql["Reputation risk"]
  node_trend_sql["Sentiment trends"]
  node_breakdown_sql["Sentiment breakdown"]
  node_company_sql["Company breakdown"]
  node_breaking_sql["Breaking news alert"]
  node_spike_sql["Negative spike detection"]
end

subgraph group_delivery["BI and alerting"]
  node_power_bi["Power BI"]
  node_office365["Office 365 email"]
end

node_risk_manager(("Risk manager"))
node_tagesschau["Tagesschau API"]
node_bert_model["German BERT"]

node_pipeline -->|"dispatches"| node_bronze_nb
node_pipeline -->|"dispatches"| node_silver_nb
node_pipeline -->|"dispatches"| node_gold_nb
node_pipeline -->|"dispatches"| node_star_nb
node_tagesschau -->|"provides articles"| node_bronze_nb
node_bronze_nb -->|"writes raw JSON"| node_bronze_store
node_silver_nb -->|"reads archive"| node_bronze_store
node_silver_nb -->|"writes cleaned articles"| node_silver_store
node_gold_nb -->|"reads articles"| node_silver_store
node_gold_nb -->|"uses aliases"| node_company_aliases
node_gold_nb -->|"scores sentiment"| node_bert_model
node_gold_nb -->|"writes analytics"| node_gold_store
node_star_nb -->|"reads analytics"| node_gold_store
node_star_nb -->|"writes dimensions"| node_star_store
node_star_store -->|"serves model"| node_power_bi
node_gold_nb -->|"returns alert text"| node_pipeline
node_pipeline -->|"sends notifications"| node_office365
node_pipeline -->|"notifies failures"| node_risk_manager
node_office365 -->|"delivers alerts"| node_risk_manager

click node_bronze_nb "https://github.com/charankumarkharade/dax40-sentiment-monitor/blob/main/notebooks/Ingestion_Bronze.ipynb"
click node_silver_nb "https://github.com/charankumarkharade/dax40-sentiment-monitor/blob/main/notebooks/Cleaning_Silver.ipynb"
click node_gold_nb "https://github.com/charankumarkharade/dax40-sentiment-monitor/blob/main/notebooks/Polished_Gold.ipynb"
click node_company_aliases "https://github.com/charankumarkharade/dax40-sentiment-monitor/blob/main/config/dax40_companies.py"
click node_star_nb "https://github.com/charankumarkharade/dax40-sentiment-monitor/blob/main/notebooks/Star_models.ipynb"
click node_reputation_sql "https://github.com/charankumarkharade/dax40-sentiment-monitor/blob/main/sql/01_reputation_risk_matrix.sql"
click node_trend_sql "https://github.com/charankumarkharade/dax40-sentiment-monitor/blob/main/sql/02_sentiment_trend.sql"
click node_breakdown_sql "https://github.com/charankumarkharade/dax40-sentiment-monitor/blob/main/sql/03_sentiment_breakdown.sql"
click node_company_sql "https://github.com/charankumarkharade/dax40-sentiment-monitor/blob/main/sql/04_company_daily_breakdown.sql"
click node_breaking_sql "https://github.com/charankumarkharade/dax40-sentiment-monitor/blob/main/sql/05_breaking_news_alert.sql"
click node_spike_sql "https://github.com/charankumarkharade/dax40-sentiment-monitor/blob/main/sql/06_negative_spike_detection.sql"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_pipeline,node_bronze_nb toneBlue
class node_bronze_store,node_silver_nb,node_silver_store,node_gold_nb,node_company_aliases,node_gold_store toneAmber
class node_star_nb,node_star_store,node_reputation_sql,node_trend_sql,node_breakdown_sql,node_company_sql,node_breaking_sql,node_spike_sql,node_tagesschau toneMint
class node_power_bi,node_office365 toneRose
class node_risk_manager,node_bert_model toneIndigo
```

## Components

### Ingestion and orchestration

- **Fabric pipeline**
- [Bronze ingestion](https://github.com/charankumarkharade/dax40-sentiment-monitor/blob/main/notebooks/Ingestion_Bronze.ipynb): `notebooks/Ingestion_Bronze.ipynb`

### Medallion data stages

- **Bronze archive**
- [Silver cleaning](https://github.com/charankumarkharade/dax40-sentiment-monitor/blob/main/notebooks/Cleaning_Silver.ipynb): `notebooks/Cleaning_Silver.ipynb`
- **Silver articles**
- [Gold enrichment](https://github.com/charankumarkharade/dax40-sentiment-monitor/blob/main/notebooks/Polished_Gold.ipynb): `notebooks/Polished_Gold.ipynb`
- [Company aliases](https://github.com/charankumarkharade/dax40-sentiment-monitor/blob/main/config/dax40_companies.py): `config/dax40_companies.py`
- **Gold analytics**

### Analytics and risk

- [Star modeling](https://github.com/charankumarkharade/dax40-sentiment-monitor/blob/main/notebooks/Star_models.ipynb): `notebooks/Star_models.ipynb`
- **Star schema**
- [Reputation risk](https://github.com/charankumarkharade/dax40-sentiment-monitor/blob/main/sql/01_reputation_risk_matrix.sql): `sql/01_reputation_risk_matrix.sql`
- [Sentiment trends](https://github.com/charankumarkharade/dax40-sentiment-monitor/blob/main/sql/02_sentiment_trend.sql): `sql/02_sentiment_trend.sql`
- [Sentiment breakdown](https://github.com/charankumarkharade/dax40-sentiment-monitor/blob/main/sql/03_sentiment_breakdown.sql): `sql/03_sentiment_breakdown.sql`
- [Company breakdown](https://github.com/charankumarkharade/dax40-sentiment-monitor/blob/main/sql/04_company_daily_breakdown.sql): `sql/04_company_daily_breakdown.sql`
- [Breaking news alert](https://github.com/charankumarkharade/dax40-sentiment-monitor/blob/main/sql/05_breaking_news_alert.sql): `sql/05_breaking_news_alert.sql`
- [Negative spike detection](https://github.com/charankumarkharade/dax40-sentiment-monitor/blob/main/sql/06_negative_spike_detection.sql): `sql/06_negative_spike_detection.sql`

### BI and alerting

- **Power BI**
- **Office 365 email**

### Other components

- **Risk manager**
- **Tagesschau API**
- **German BERT**

## Connections

- Fabric pipeline → Bronze ingestion: dispatches
- Fabric pipeline → Silver cleaning: dispatches
- Fabric pipeline → Gold enrichment: dispatches
- Fabric pipeline → Star modeling: dispatches
- Tagesschau API → Bronze ingestion: provides articles
- Bronze ingestion → Bronze archive: writes raw JSON
- Silver cleaning → Bronze archive: reads archive
- Silver cleaning → Silver articles: writes cleaned articles
- Gold enrichment → Silver articles: reads articles
- Gold enrichment → Company aliases: uses aliases
- Gold enrichment → German BERT: scores sentiment
- Gold enrichment → Gold analytics: writes analytics
- Star modeling → Gold analytics: reads analytics
- Star modeling → Star schema: writes dimensions
- Star schema → Power BI: serves model
- Gold enrichment → Fabric pipeline: returns alert text
- Fabric pipeline → Office 365 email: sends notifications
- Fabric pipeline → Risk manager: notifies failures
- Office 365 email → Risk manager: delivers alerts
