How addetsi/devinsights works
DevInsights collects public GitHub repository activity, turns it into compliance and DORA analytics, and presents dashboards and AI-generated reports. The visible runtime flow starts with configured scraping and Kafka publication, then Kafka consumption into blob or file landing storage; a separate loader and dbt project support warehouse transformation. The README confirms Azure Event Hubs, Blob Storage, Azure SQL, and medallion transformations, but their complete wiring is not shown. Edges without direct source evidence are omitted; documented platform relationships cite the README.
Main entry points: scraper.py Scraper engine · db_connection.py SQL access · report_generator.py Report workflow · consumer.py Kafka consumer
Components 27
Collection & Streaming
- Scraper engine
scrapers/src/scraper.py - Scrape configuration
scrapers/src/config.py - GitHub API
- Message validation
scrapers/src/validators.py - Message schemas
scrapers/src/schemas.py - Kafka producer
scrapers/src/producer.py - Event Hubs broker
- Kafka consumer
consumer/src/consumer.py - Blob sink
consumer/src/blob_sink.py - File sink
consumer/src/sink.py
Storage & Analytics
- Azure Blob Storage
- Blob-to-SQL loader
loader/src/load_blob_to_sql.py - Bronze models
dbt_project/models/bronze - Silver models
dbt_project/models/silver - Gold analytics
dbt_project/models/gold - Azure SQL
ML & Dashboard
- Anomaly training
ml/train_anomaly_model.py - Anomaly scoring
ml/predict.py - Metrics dashboard
streamlit_app/app.py
AI Reporting
- Report workflow
streamlit_app/pages/report_generator.py - Report aggregation
streamlit_app/utils/report_data.py - Claude narrative
streamlit_app/utils/claude_client.py - Word document builder
streamlit_app/utils/doc_generator.py - Anthropic API
Shared Services
- SQL access
common/db_connection.py - Logging configuration
common/logging_config.py
Other components
- Platform user
Connections 24
- Platform user → Scraper engine: starts scrape
scrapers/src/scraper.py - Scraper engine → Scrape configuration: loads config
scrapers/src/scraper.py - Scraper engine → GitHub API: fetches endpoints
scrapers/src/scraper.py - Scraper engine → Message validation: validates items
scrapers/src/scraper.py - Message validation → Message schemas: checks schema
scrapers/src/validators.py - Scraper engine → Kafka producer: publishes messages
scrapers/src/scraper.py - Kafka producer → Event Hubs broker: sends events
README.md - Event Hubs broker → Kafka consumer: delivers events
README.md - Kafka consumer → Blob sink: selects sink
consumer/src/consumer.py - Kafka consumer → File sink: selects fallback
consumer/src/consumer.py - Blob sink → Azure Blob Storage: writes blobs
README.md - Azure Blob Storage → Blob-to-SQL loader: loads landing data
- Blob-to-SQL loader → Azure SQL: loads warehouse
- Bronze models → Silver models: medallion transform
README.md - Silver models → Gold analytics: medallion transform
README.md - Gold analytics → Azure SQL: stores analytics
- Anomaly training → SQL access: queries activity
ml/train_anomaly_model.py - Anomaly scoring → SQL access: queries data
ml/predict.py - Metrics dashboard → SQL access: queries metrics
streamlit_app/app.py - Report workflow → Report aggregation: builds summaries
streamlit_app/pages/report_generator.py - Report workflow → Claude narrative: requests narrative
streamlit_app/pages/report_generator.py - Claude narrative → Anthropic API: generates text
streamlit_app/utils/claude_client.py - Report workflow → Word document builder: builds document
streamlit_app/pages/report_generator.py - Report aggregation → SQL access: queries gold data
streamlit_app/utils/report_data.py