How databrickslabs/dqx works
DQX is a PySpark data-quality platform: users define or load checks, apply them to batch or streaming data, and inspect failures and metrics. Its main runtime centers on the engine, with distinct rule storage, profiling, anomaly detection, and action/alerting capabilities. The repository also includes DQX Studio and an MCP server as user-facing products. Some links are documented rather than visible in sampled code; those are marked dashed with no evidence path.
Main entry points: engine.py DQ Engine · checks_storage.py Checks Storage · profiler.py Data Profiler · dq_action.py Actions and Alerts
Components 24
Quality Runtime
- DQ Engine
src/databricks/labs/dqx/engine.py - Run Config
src/databricks/labs/dqx/config.py - Rule Definitions
src/databricks/labs/dqx/rule.py - Checks Storage
src/databricks/labs/dqx/checks_storage.py - Data I/O
src/databricks/labs/dqx/io.py - Checks Serialization
src/databricks/labs/dqx/checks_serializer.py - Checks Validation
src/databricks/labs/dqx/checks_validator.py - Rule Fingerprints
src/databricks/labs/dqx/rule_fingerprint.py
Rules and Discovery
- Data Profiler
src/databricks/labs/dqx/profiler/profiler.py - Rule Generator
src/databricks/labs/dqx/profiler/generator.py - LLM Rules
src/databricks/labs/dqx/llm/llm_engine.py - Anomaly Detection
src/databricks/labs/dqx/anomaly/anomaly_engine.py
Results and Actions
- Quality Metrics
src/databricks/labs/dqx/metrics_observer.py - Actions and Alerts
src/databricks/labs/dqx/actions/dq_action.py - Action Event Store
src/databricks/labs/dqx/actions/event_storage.py
DQX Studio
- Studio Interface
app/src/databricks_labs_dqx_app/ui - Studio API
app/src/databricks_labs_dqx_app/backend/routes/v1 - Studio Services
app/src/databricks_labs_dqx_app/backend/services - Scheduled Runner
app/tasks/src/dqx_task_runner/runner.py
Agent Integration
- MCP Server
mcp-server/server/app.py - MCP Tools
mcp-server/server/tools.py - MCP Runner
mcp-server/runner/src/dqx_mcp_runner/runner.py
Other components
- Data User
- Spark Data
Connections 24
- Data User → DQ Engine: runs checks
README.md - Spark Data → DQ Engine: validates
README.md - DQ Engine → Rule Definitions: uses rules
src/databricks/labs/dqx/engine.py - DQ Engine → Run Config: uses config
src/databricks/labs/dqx/engine.py - DQ Engine → Checks Storage: loads checks
src/databricks/labs/dqx/engine.py - Checks Storage → Checks Serialization: serializes checks
src/databricks/labs/dqx/checks_storage.py - Checks Storage → Checks Validation: validates checks
src/databricks/labs/dqx/checks_storage.py - Checks Storage → Rule Fingerprints: fingerprints rules
src/databricks/labs/dqx/checks_storage.py - DQ Engine → Data I/O: reads and writes
src/databricks/labs/dqx/engine.py - Data User → Data Profiler: profiles data
README.md - Data Profiler → Rule Generator: suggests rules
README.md - Data User → LLM Rules: requests suggestions
README.md - Data User → Anomaly Detection: detects anomalies
README.md - DQ Engine → Quality Metrics: reports metrics
README.md - DQ Engine → Actions and Alerts: triggers actions
README.md - Actions and Alerts → Action Event Store: stores events
src/databricks/labs/dqx/actions/event_storage.py - Data User → Studio Interface: authors and reviews
README.md - Studio Interface → Studio API: calls API
- Studio API → Studio Services: dispatches requests
- Studio Services → DQ Engine: runs quality checks
- Scheduled Runner → DQ Engine: runs scheduled checks
app/tasks/src/dqx_task_runner/runner.py - Data User → MCP Server: connects agent
README.md - MCP Server → MCP Tools: loads tools
mcp-server/server/app.py - MCP Tools → MCP Runner: invokes DQX