How nirvishagarara/ai-data-quality-copilot works
Data Quality Copilot monitors a DuckDB warehouse for schema drift and statistical anomalies, generates dbt-compatible tests, and can explain detected issues with Claude, alert Slack, and render lineage. Users enter through the CLI or Streamlit dashboard; the CLI dispatches commands to the monitoring and analysis modules, while the dashboard can also access the FastAPI service. Results and historical baselines are persisted for review. The README documents the principal stages and optional capabilities; some implementation details, including test generation and portions of the API and dashboard, were not sampled, so their connections are limited to documented or indexed evidence.
Main entry points: dq_copilot.py CLI Commands · schema_monitor.py Schema Drift Monitor · anomaly_detector.py Anomaly Detector · main.py REST API
Components 16
User Interfaces
- CLI Commands
dq_copilot.py - Streamlit Dashboard
frontend/app.py - REST API
src/api/main.py
Warehouse Observability
- Schema Drift Monitor
src/monitoring/schema_monitor.py - Anomaly Detector
src/monitoring/anomaly_detector.py - Test Generator
src/monitoring/test_generator.py
Analysis and Delivery
- Root-Cause Analyzer
src/llm/root_cause_analyzer.py - Claude API
- Slack Webhook
- Slack Alerts
src/alerts/slack_alerts.py - Lineage Graph
src/lineage/lineage_graph.py
Configuration and State
- DuckDB Warehouse
data/warehouse.duckdb - Schema and Metric History
data/snapshots - Runtime Configuration
src/config.py - Reports and Test Outputs
data
Other components
- Engineer
Connections 24
- Engineer → CLI Commands: runs commands
README.md - Engineer → Streamlit Dashboard: explores results
README.md - CLI Commands → Schema Drift Monitor: runs scan
dq_copilot.py - CLI Commands → Anomaly Detector: runs scan
dq_copilot.py - CLI Commands → Root-Cause Analyzer: dispatches explain
dq_copilot.py - CLI Commands → Test Generator: dispatches tests
dq_copilot.py - CLI Commands → Slack Alerts: dispatches alerts
dq_copilot.py - CLI Commands → Lineage Graph: dispatches lineage
dq_copilot.py - Streamlit Dashboard → REST API: requests data
src/api/main.py - REST API → Anomaly Detector: runs detector
src/api/main.py - REST API → Schema Drift Monitor: serves schema
src/api/main.py - Schema Drift Monitor → DuckDB Warehouse: reads schema
src/monitoring/schema_monitor.py - Anomaly Detector → DuckDB Warehouse: measures tables
src/monitoring/anomaly_detector.py - Schema Drift Monitor → Schema and Metric History: stores drift history
src/monitoring/schema_monitor.py - Anomaly Detector → Schema and Metric History: reads baselines
README.md - Test Generator → DuckDB Warehouse: profiles tables
README.md - Schema Drift Monitor → Runtime Configuration: loads settings
src/monitoring/schema_monitor.py - Anomaly Detector → Runtime Configuration: loads settings
src/monitoring/anomaly_detector.py - Root-Cause Analyzer → Claude API: requests analysis
src/llm/root_cause_analyzer.py - Root-Cause Analyzer → Reports and Test Outputs: saves reports
src/llm/root_cause_analyzer.py - Slack Alerts → Slack Webhook: posts alerts
src/alerts/slack_alerts.py - Lineage Graph → Reports and Test Outputs: uses issue status
src/lineage/lineage_graph.py - Lineage Graph → Runtime Configuration: loads graph edges
src/lineage/lineage_graph.py - Test Generator → Reports and Test Outputs: writes generated tests
README.md