# databrickslabs/dqx architecture

> Architecture diagram of the GitHub repository databrickslabs/dqx, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-07). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/databrickslabs/dqx
- Repository: https://github.com/databrickslabs/dqx

## Overview

DQX is a PySpark data-quality platform: users define or load checks, apply them to batch or streaming data, and inspect failures and metrics. Its main runtime centers on the engine, with distinct rule storage, profiling, anomaly detection, and action/alerting capabilities. The repository also includes DQX Studio and an MCP server as user-facing products. Some links are documented rather than visible in sampled code; those are marked dashed with no evidence path.

Main entry points: [`src/databricks/labs/dqx/engine.py`](https://github.com/databrickslabs/dqx/blob/main/src/databricks/labs/dqx/engine.py) (DQ Engine), [`src/databricks/labs/dqx/checks_storage.py`](https://github.com/databrickslabs/dqx/blob/main/src/databricks/labs/dqx/checks_storage.py) (Checks Storage), [`src/databricks/labs/dqx/profiler/profiler.py`](https://github.com/databrickslabs/dqx/blob/main/src/databricks/labs/dqx/profiler/profiler.py) (Data Profiler), [`src/databricks/labs/dqx/actions/dq_action.py`](https://github.com/databrickslabs/dqx/blob/main/src/databricks/labs/dqx/actions/dq_action.py) (Actions and Alerts)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_runtime["Quality Runtime"]
  node_engine["DQ Engine<br/>[engine.py]"]
  node_config["Run Config<br/>[config.py]"]
  node_rules["Rule Definitions<br/>[rule.py]"]
  node_checks_storage["Checks Storage<br/>[checks_storage.py]"]
  node_io["Data I/O<br/>[io.py]"]
  node_checks_serialization["Checks Serialization"]
  node_checks_validation["Checks Validation"]
  node_fingerprint["Rule Fingerprints"]
end

subgraph group_authoring["Rules and Discovery"]
  node_profiler["Data Profiler<br/>[profiler.py]"]
  node_generator["Rule Generator<br/>[generator.py]"]
  node_llm["LLM Rules<br/>[llm_engine.py]"]
  node_anomaly["Anomaly Detection<br/>[anomaly_engine.py]"]
end

subgraph group_operations["Results and Actions"]
  node_metrics["Quality Metrics"]
  node_actions["Actions and Alerts<br/>[dq_action.py]"]
  node_event_store[("Action Event Store<br/>[event_storage.py]")]
end

subgraph group_studio["DQX Studio"]
  node_studio_ui["Studio Interface"]
  node_studio_api["Studio API"]
  node_studio_services["Studio Services"]
  node_studio_runner["Scheduled Runner<br/>[runner.py]"]
end

subgraph group_agents["Agent Integration"]
  node_mcp_server["MCP Server<br/>[app.py]"]
  node_mcp_tools["MCP Tools<br/>[tools.py]"]
  node_mcp_runner["MCP Runner<br/>[runner.py]"]
end

node_consumer(("Data User"))
node_data[("Spark Data")]

node_consumer -.->|"runs checks"| node_engine
node_data -.->|"validates"| node_engine
node_engine -->|"uses rules"| node_rules
node_engine -->|"uses config"| node_config
node_engine -->|"loads checks"| node_checks_storage
node_checks_storage -->|"serializes checks"| node_checks_serialization
node_checks_storage -->|"validates checks"| node_checks_validation
node_checks_storage -->|"fingerprints rules"| node_fingerprint
node_engine -->|"reads and writes"| node_io
node_consumer -.->|"profiles data"| node_profiler
node_profiler -.->|"suggests rules"| node_generator
node_consumer -.->|"requests suggestions"| node_llm
node_consumer -.->|"detects anomalies"| node_anomaly
node_engine -.->|"reports metrics"| node_metrics
node_engine -.->|"triggers actions"| node_actions
node_actions -->|"stores events"| node_event_store
node_consumer -.->|"authors and reviews"| node_studio_ui
node_studio_ui -.->|"calls API"| node_studio_api
node_studio_api -.->|"dispatches requests"| node_studio_services
node_studio_services -.->|"runs quality checks"| node_engine
node_studio_runner -->|"runs scheduled checks"| node_engine
node_consumer -.->|"connects agent"| node_mcp_server
node_mcp_server -->|"loads tools"| node_mcp_tools
node_mcp_tools -.->|"invokes DQX"| node_mcp_runner

click node_engine "https://github.com/databrickslabs/dqx/blob/main/src/databricks/labs/dqx/engine.py"
click node_config "https://github.com/databrickslabs/dqx/blob/main/src/databricks/labs/dqx/config.py"
click node_rules "https://github.com/databrickslabs/dqx/blob/main/src/databricks/labs/dqx/rule.py"
click node_checks_storage "https://github.com/databrickslabs/dqx/blob/main/src/databricks/labs/dqx/checks_storage.py"
click node_io "https://github.com/databrickslabs/dqx/blob/main/src/databricks/labs/dqx/io.py"
click node_checks_serialization "https://github.com/databrickslabs/dqx/blob/main/src/databricks/labs/dqx/checks_serializer.py"
click node_checks_validation "https://github.com/databrickslabs/dqx/blob/main/src/databricks/labs/dqx/checks_validator.py"
click node_fingerprint "https://github.com/databrickslabs/dqx/blob/main/src/databricks/labs/dqx/rule_fingerprint.py"
click node_profiler "https://github.com/databrickslabs/dqx/blob/main/src/databricks/labs/dqx/profiler/profiler.py"
click node_generator "https://github.com/databrickslabs/dqx/blob/main/src/databricks/labs/dqx/profiler/generator.py"
click node_llm "https://github.com/databrickslabs/dqx/blob/main/src/databricks/labs/dqx/llm/llm_engine.py"
click node_anomaly "https://github.com/databrickslabs/dqx/blob/main/src/databricks/labs/dqx/anomaly/anomaly_engine.py"
click node_metrics "https://github.com/databrickslabs/dqx/blob/main/src/databricks/labs/dqx/metrics_observer.py"
click node_actions "https://github.com/databrickslabs/dqx/blob/main/src/databricks/labs/dqx/actions/dq_action.py"
click node_event_store "https://github.com/databrickslabs/dqx/blob/main/src/databricks/labs/dqx/actions/event_storage.py"
click node_studio_ui "https://github.com/databrickslabs/dqx/tree/main/app/src/databricks_labs_dqx_app/ui"
click node_studio_api "https://github.com/databrickslabs/dqx/tree/main/app/src/databricks_labs_dqx_app/backend/routes/v1"
click node_studio_services "https://github.com/databrickslabs/dqx/tree/main/app/src/databricks_labs_dqx_app/backend/services"
click node_studio_runner "https://github.com/databrickslabs/dqx/blob/main/app/tasks/src/dqx_task_runner/runner.py"
click node_mcp_server "https://github.com/databrickslabs/dqx/blob/main/mcp-server/server/app.py"
click node_mcp_tools "https://github.com/databrickslabs/dqx/blob/main/mcp-server/server/tools.py"
click node_mcp_runner "https://github.com/databrickslabs/dqx/blob/main/mcp-server/runner/src/dqx_mcp_runner/runner.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_engine,node_config,node_rules,node_checks_storage,node_io,node_checks_serialization,node_checks_validation,node_fingerprint,node_consumer toneBlue
class node_profiler,node_generator,node_llm,node_anomaly,node_data toneAmber
class node_metrics,node_actions,node_event_store toneMint
class node_studio_ui,node_studio_api,node_studio_services,node_studio_runner toneRose
class node_mcp_server,node_mcp_tools,node_mcp_runner toneIndigo
```

## Components

### Quality Runtime

- [DQ Engine](https://github.com/databrickslabs/dqx/blob/main/src/databricks/labs/dqx/engine.py): `src/databricks/labs/dqx/engine.py`
- [Run Config](https://github.com/databrickslabs/dqx/blob/main/src/databricks/labs/dqx/config.py): `src/databricks/labs/dqx/config.py`
- [Rule Definitions](https://github.com/databrickslabs/dqx/blob/main/src/databricks/labs/dqx/rule.py): `src/databricks/labs/dqx/rule.py`
- [Checks Storage](https://github.com/databrickslabs/dqx/blob/main/src/databricks/labs/dqx/checks_storage.py): `src/databricks/labs/dqx/checks_storage.py`
- [Data I/O](https://github.com/databrickslabs/dqx/blob/main/src/databricks/labs/dqx/io.py): `src/databricks/labs/dqx/io.py`
- [Checks Serialization](https://github.com/databrickslabs/dqx/blob/main/src/databricks/labs/dqx/checks_serializer.py): `src/databricks/labs/dqx/checks_serializer.py`
- [Checks Validation](https://github.com/databrickslabs/dqx/blob/main/src/databricks/labs/dqx/checks_validator.py): `src/databricks/labs/dqx/checks_validator.py`
- [Rule Fingerprints](https://github.com/databrickslabs/dqx/blob/main/src/databricks/labs/dqx/rule_fingerprint.py): `src/databricks/labs/dqx/rule_fingerprint.py`

### Rules and Discovery

- [Data Profiler](https://github.com/databrickslabs/dqx/blob/main/src/databricks/labs/dqx/profiler/profiler.py): `src/databricks/labs/dqx/profiler/profiler.py`
- [Rule Generator](https://github.com/databrickslabs/dqx/blob/main/src/databricks/labs/dqx/profiler/generator.py): `src/databricks/labs/dqx/profiler/generator.py`
- [LLM Rules](https://github.com/databrickslabs/dqx/blob/main/src/databricks/labs/dqx/llm/llm_engine.py): `src/databricks/labs/dqx/llm/llm_engine.py`
- [Anomaly Detection](https://github.com/databrickslabs/dqx/blob/main/src/databricks/labs/dqx/anomaly/anomaly_engine.py): `src/databricks/labs/dqx/anomaly/anomaly_engine.py`

### Results and Actions

- [Quality Metrics](https://github.com/databrickslabs/dqx/blob/main/src/databricks/labs/dqx/metrics_observer.py): `src/databricks/labs/dqx/metrics_observer.py`
- [Actions and Alerts](https://github.com/databrickslabs/dqx/blob/main/src/databricks/labs/dqx/actions/dq_action.py): `src/databricks/labs/dqx/actions/dq_action.py`
- [Action Event Store](https://github.com/databrickslabs/dqx/blob/main/src/databricks/labs/dqx/actions/event_storage.py): `src/databricks/labs/dqx/actions/event_storage.py`

### DQX Studio

- [Studio Interface](https://github.com/databrickslabs/dqx/tree/main/app/src/databricks_labs_dqx_app/ui): `app/src/databricks_labs_dqx_app/ui`
- [Studio API](https://github.com/databrickslabs/dqx/tree/main/app/src/databricks_labs_dqx_app/backend/routes/v1): `app/src/databricks_labs_dqx_app/backend/routes/v1`
- [Studio Services](https://github.com/databrickslabs/dqx/tree/main/app/src/databricks_labs_dqx_app/backend/services): `app/src/databricks_labs_dqx_app/backend/services`
- [Scheduled Runner](https://github.com/databrickslabs/dqx/blob/main/app/tasks/src/dqx_task_runner/runner.py): `app/tasks/src/dqx_task_runner/runner.py`

### Agent Integration

- [MCP Server](https://github.com/databrickslabs/dqx/blob/main/mcp-server/server/app.py): `mcp-server/server/app.py`
- [MCP Tools](https://github.com/databrickslabs/dqx/blob/main/mcp-server/server/tools.py): `mcp-server/server/tools.py`
- [MCP Runner](https://github.com/databrickslabs/dqx/blob/main/mcp-server/runner/src/dqx_mcp_runner/runner.py): `mcp-server/runner/src/dqx_mcp_runner/runner.py`

### Other components

- **Data User**
- **Spark Data**

## Connections

- Data User → DQ Engine: runs checks (evidence: `README.md`)
- Spark Data → DQ Engine: validates (evidence: `README.md`)
- DQ Engine → Rule Definitions: uses rules (evidence: `src/databricks/labs/dqx/engine.py`)
- DQ Engine → Run Config: uses config (evidence: `src/databricks/labs/dqx/engine.py`)
- DQ Engine → Checks Storage: loads checks (evidence: `src/databricks/labs/dqx/engine.py`)
- Checks Storage → Checks Serialization: serializes checks (evidence: `src/databricks/labs/dqx/checks_storage.py`)
- Checks Storage → Checks Validation: validates checks (evidence: `src/databricks/labs/dqx/checks_storage.py`)
- Checks Storage → Rule Fingerprints: fingerprints rules (evidence: `src/databricks/labs/dqx/checks_storage.py`)
- DQ Engine → Data I/O: reads and writes (evidence: `src/databricks/labs/dqx/engine.py`)
- Data User → Data Profiler: profiles data (evidence: `README.md`)
- Data Profiler → Rule Generator: suggests rules (evidence: `README.md`)
- Data User → LLM Rules: requests suggestions (evidence: `README.md`)
- Data User → Anomaly Detection: detects anomalies (evidence: `README.md`)
- DQ Engine → Quality Metrics: reports metrics (evidence: `README.md`)
- DQ Engine → Actions and Alerts: triggers actions (evidence: `README.md`)
- Actions and Alerts → Action Event Store: stores events (evidence: `src/databricks/labs/dqx/actions/event_storage.py`)
- Data User → Studio Interface: authors and reviews (evidence: `README.md`)
- Studio Interface → Studio API: calls API
- Studio API → Studio Services: dispatches requests
- Studio Services → DQ Engine: runs quality checks
- Scheduled Runner → DQ Engine: runs scheduled checks (evidence: `app/tasks/src/dqx_task_runner/runner.py`)
- Data User → MCP Server: connects agent (evidence: `README.md`)
- MCP Server → MCP Tools: loads tools (evidence: `mcp-server/server/app.py`)
- MCP Tools → MCP Runner: invokes DQX
