# addetsi/devinsights architecture

> Architecture diagram of the GitHub repository addetsi/devinsights, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-21). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/addetsi/devinsights
- Repository: https://github.com/addetsi/devinsights

## Overview

DevInsights collects public GitHub repository activity, turns it into compliance and DORA analytics, and presents dashboards and AI-generated reports. The visible runtime flow starts with configured scraping and Kafka publication, then Kafka consumption into blob or file landing storage; a separate loader and dbt project support warehouse transformation. The README confirms Azure Event Hubs, Blob Storage, Azure SQL, and medallion transformations, but their complete wiring is not shown. Edges without direct source evidence are omitted; documented platform relationships cite the README.

Main entry points: [`scrapers/src/scraper.py`](https://github.com/addetsi/devinsights/blob/main/scrapers/src/scraper.py) (Scraper engine), [`common/db_connection.py`](https://github.com/addetsi/devinsights/blob/main/common/db_connection.py) (SQL access), [`streamlit_app/pages/report_generator.py`](https://github.com/addetsi/devinsights/blob/main/streamlit_app/pages/report_generator.py) (Report workflow), [`consumer/src/consumer.py`](https://github.com/addetsi/devinsights/blob/main/consumer/src/consumer.py) (Kafka consumer)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_collect["Collection &amp; Streaming"]
  node_scraper["Scraper engine<br/>[scraper.py]"]
  node_scrape_config["Scrape configuration<br/>[config.py]"]
  node_github["GitHub API"]
  node_validators["Message validation<br/>[validators.py]"]
  node_schemas["Message schemas<br/>[schemas.py]"]
  node_producer["Kafka producer<br/>[producer.py]"]
  node_broker["Event Hubs broker"]
  node_consumer["Kafka consumer<br/>[consumer.py]"]
  node_blob_sink["Blob sink<br/>[blob_sink.py]"]
  node_file_sink["File sink<br/>[sink.py]"]
end

subgraph group_warehouse["Storage &amp; Analytics"]
  node_blob[("Azure Blob Storage")]
  node_loader["Blob-to-SQL loader"]
  node_bronze["Bronze models"]
  node_silver["Silver models"]
  node_gold["Gold analytics"]
  node_sql[("Azure SQL")]
end

subgraph group_insights["ML &amp; Dashboard"]
  node_anomaly_train["Anomaly training"]
  node_anomaly_predict["Anomaly scoring<br/>[predict.py]"]
  node_dashboard["Metrics dashboard<br/>[app.py]"]
end

subgraph group_reports["AI Reporting"]
  node_report_page["Report workflow"]
  node_report_data["Report aggregation<br/>[report_data.py]"]
  node_claude["Claude narrative<br/>[claude_client.py]"]
  node_docx["Word document builder<br/>[doc_generator.py]"]
  node_anthropic["Anthropic API"]
end

subgraph group_shared["Shared Services"]
  node_db_access["SQL access<br/>[db_connection.py]"]
  node_logging["Logging configuration<br/>[logging_config.py]"]
end

node_operator(("Platform user"))

node_operator -->|"starts scrape"| node_scraper
node_scraper -->|"loads config"| node_scrape_config
node_scraper -->|"fetches endpoints"| node_github
node_scraper -->|"validates items"| node_validators
node_validators -->|"checks schema"| node_schemas
node_scraper -->|"publishes messages"| node_producer
node_producer -->|"sends events"| node_broker
node_broker -->|"delivers events"| node_consumer
node_consumer -->|"selects sink"| node_blob_sink
node_consumer -->|"selects fallback"| node_file_sink
node_blob_sink -->|"writes blobs"| node_blob
node_blob -.->|"loads landing data"| node_loader
node_loader -.->|"loads warehouse"| node_sql
node_bronze -->|"medallion transform"| node_silver
node_silver -->|"medallion transform"| node_gold
node_gold -.->|"stores analytics"| node_sql
node_anomaly_train -->|"queries activity"| node_db_access
node_anomaly_predict -->|"queries data"| node_db_access
node_dashboard -->|"queries metrics"| node_db_access
node_report_page -->|"builds summaries"| node_report_data
node_report_page -->|"requests narrative"| node_claude
node_claude -->|"generates text"| node_anthropic
node_report_page -->|"builds document"| node_docx
node_report_data -->|"queries gold data"| node_db_access

click node_scraper "https://github.com/addetsi/devinsights/blob/main/scrapers/src/scraper.py"
click node_scrape_config "https://github.com/addetsi/devinsights/blob/main/scrapers/src/config.py"
click node_validators "https://github.com/addetsi/devinsights/blob/main/scrapers/src/validators.py"
click node_schemas "https://github.com/addetsi/devinsights/blob/main/scrapers/src/schemas.py"
click node_producer "https://github.com/addetsi/devinsights/blob/main/scrapers/src/producer.py"
click node_consumer "https://github.com/addetsi/devinsights/blob/main/consumer/src/consumer.py"
click node_blob_sink "https://github.com/addetsi/devinsights/blob/main/consumer/src/blob_sink.py"
click node_file_sink "https://github.com/addetsi/devinsights/blob/main/consumer/src/sink.py"
click node_loader "https://github.com/addetsi/devinsights/blob/main/loader/src/load_blob_to_sql.py"
click node_bronze "https://github.com/addetsi/devinsights/tree/main/dbt_project/models/bronze"
click node_silver "https://github.com/addetsi/devinsights/tree/main/dbt_project/models/silver"
click node_gold "https://github.com/addetsi/devinsights/tree/main/dbt_project/models/gold"
click node_db_access "https://github.com/addetsi/devinsights/blob/main/common/db_connection.py"
click node_anomaly_train "https://github.com/addetsi/devinsights/blob/main/ml/train_anomaly_model.py"
click node_anomaly_predict "https://github.com/addetsi/devinsights/blob/main/ml/predict.py"
click node_dashboard "https://github.com/addetsi/devinsights/blob/main/streamlit_app/app.py"
click node_report_page "https://github.com/addetsi/devinsights/blob/main/streamlit_app/pages/report_generator.py"
click node_report_data "https://github.com/addetsi/devinsights/blob/main/streamlit_app/utils/report_data.py"
click node_claude "https://github.com/addetsi/devinsights/blob/main/streamlit_app/utils/claude_client.py"
click node_docx "https://github.com/addetsi/devinsights/blob/main/streamlit_app/utils/doc_generator.py"
click node_logging "https://github.com/addetsi/devinsights/blob/main/common/logging_config.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_scraper,node_scrape_config,node_github,node_validators,node_schemas,node_producer,node_broker,node_consumer,node_blob_sink,node_file_sink,node_operator toneBlue
class node_blob,node_loader,node_bronze,node_silver,node_gold,node_sql toneAmber
class node_anomaly_train,node_anomaly_predict,node_dashboard toneMint
class node_report_page,node_report_data,node_claude,node_docx,node_anthropic toneRose
class node_db_access,node_logging toneIndigo
```

## Components

### Collection & Streaming

- [Scraper engine](https://github.com/addetsi/devinsights/blob/main/scrapers/src/scraper.py): `scrapers/src/scraper.py`
- [Scrape configuration](https://github.com/addetsi/devinsights/blob/main/scrapers/src/config.py): `scrapers/src/config.py`
- **GitHub API**
- [Message validation](https://github.com/addetsi/devinsights/blob/main/scrapers/src/validators.py): `scrapers/src/validators.py`
- [Message schemas](https://github.com/addetsi/devinsights/blob/main/scrapers/src/schemas.py): `scrapers/src/schemas.py`
- [Kafka producer](https://github.com/addetsi/devinsights/blob/main/scrapers/src/producer.py): `scrapers/src/producer.py`
- **Event Hubs broker**
- [Kafka consumer](https://github.com/addetsi/devinsights/blob/main/consumer/src/consumer.py): `consumer/src/consumer.py`
- [Blob sink](https://github.com/addetsi/devinsights/blob/main/consumer/src/blob_sink.py): `consumer/src/blob_sink.py`
- [File sink](https://github.com/addetsi/devinsights/blob/main/consumer/src/sink.py): `consumer/src/sink.py`

### Storage & Analytics

- **Azure Blob Storage**
- [Blob-to-SQL loader](https://github.com/addetsi/devinsights/blob/main/loader/src/load_blob_to_sql.py): `loader/src/load_blob_to_sql.py`
- [Bronze models](https://github.com/addetsi/devinsights/tree/main/dbt_project/models/bronze): `dbt_project/models/bronze`
- [Silver models](https://github.com/addetsi/devinsights/tree/main/dbt_project/models/silver): `dbt_project/models/silver`
- [Gold analytics](https://github.com/addetsi/devinsights/tree/main/dbt_project/models/gold): `dbt_project/models/gold`
- **Azure SQL**

### ML & Dashboard

- [Anomaly training](https://github.com/addetsi/devinsights/blob/main/ml/train_anomaly_model.py): `ml/train_anomaly_model.py`
- [Anomaly scoring](https://github.com/addetsi/devinsights/blob/main/ml/predict.py): `ml/predict.py`
- [Metrics dashboard](https://github.com/addetsi/devinsights/blob/main/streamlit_app/app.py): `streamlit_app/app.py`

### AI Reporting

- [Report workflow](https://github.com/addetsi/devinsights/blob/main/streamlit_app/pages/report_generator.py): `streamlit_app/pages/report_generator.py`
- [Report aggregation](https://github.com/addetsi/devinsights/blob/main/streamlit_app/utils/report_data.py): `streamlit_app/utils/report_data.py`
- [Claude narrative](https://github.com/addetsi/devinsights/blob/main/streamlit_app/utils/claude_client.py): `streamlit_app/utils/claude_client.py`
- [Word document builder](https://github.com/addetsi/devinsights/blob/main/streamlit_app/utils/doc_generator.py): `streamlit_app/utils/doc_generator.py`
- **Anthropic API**

### Shared Services

- [SQL access](https://github.com/addetsi/devinsights/blob/main/common/db_connection.py): `common/db_connection.py`
- [Logging configuration](https://github.com/addetsi/devinsights/blob/main/common/logging_config.py): `common/logging_config.py`

### Other components

- **Platform user**

## Connections

- Platform user → Scraper engine: starts scrape (evidence: `scrapers/src/scraper.py`)
- Scraper engine → Scrape configuration: loads config (evidence: `scrapers/src/scraper.py`)
- Scraper engine → GitHub API: fetches endpoints (evidence: `scrapers/src/scraper.py`)
- Scraper engine → Message validation: validates items (evidence: `scrapers/src/scraper.py`)
- Message validation → Message schemas: checks schema (evidence: `scrapers/src/validators.py`)
- Scraper engine → Kafka producer: publishes messages (evidence: `scrapers/src/scraper.py`)
- Kafka producer → Event Hubs broker: sends events (evidence: `README.md`)
- Event Hubs broker → Kafka consumer: delivers events (evidence: `README.md`)
- Kafka consumer → Blob sink: selects sink (evidence: `consumer/src/consumer.py`)
- Kafka consumer → File sink: selects fallback (evidence: `consumer/src/consumer.py`)
- Blob sink → Azure Blob Storage: writes blobs (evidence: `README.md`)
- Azure Blob Storage → Blob-to-SQL loader: loads landing data
- Blob-to-SQL loader → Azure SQL: loads warehouse
- Bronze models → Silver models: medallion transform (evidence: `README.md`)
- Silver models → Gold analytics: medallion transform (evidence: `README.md`)
- Gold analytics → Azure SQL: stores analytics
- Anomaly training → SQL access: queries activity (evidence: `ml/train_anomaly_model.py`)
- Anomaly scoring → SQL access: queries data (evidence: `ml/predict.py`)
- Metrics dashboard → SQL access: queries metrics (evidence: `streamlit_app/app.py`)
- Report workflow → Report aggregation: builds summaries (evidence: `streamlit_app/pages/report_generator.py`)
- Report workflow → Claude narrative: requests narrative (evidence: `streamlit_app/pages/report_generator.py`)
- Claude narrative → Anthropic API: generates text (evidence: `streamlit_app/utils/claude_client.py`)
- Report workflow → Word document builder: builds document (evidence: `streamlit_app/pages/report_generator.py`)
- Report aggregation → SQL access: queries gold data (evidence: `streamlit_app/utils/report_data.py`)
