# sparkdq-community/sparkdq architecture

> Architecture diagram of the GitHub repository sparkdq-community/sparkdq, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-30). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/sparkdq-community/sparkdq
- Repository: https://github.com/sparkdq-community/sparkdq

## Overview

##### Project Type and Purpose
- SparkDQ is an open-source, Spark-native data-quality validation library for PySpark. It provides a programmatic and declarative API for defining, executing, and reporting both row-level and aggregate-level checks directly inside Spark pipelines.

##### Main Components
a. Data Source / Spark Cluster
- External systems (files, tables, streams) feed DataFrames into Spark
- The SparkSession and cluster resources (driver, executors) that run the pipeline
b. Check Definition Layer
- sparkdq.management.CheckSet
- Configuration classes in sparkdq.checks.*_config (e.g. NullCheckConfig)
- Declarative configs (YAML/JSON dictionaries) loaded via CheckSet.add_checks_from_dicts
c. Factory Layer
- sparkdq.factory.check_config_registry: Registry of available check types
- sparkdq.factory.check_factory: Instantiates concrete Check objects from configs
d. Core Check Abstractions
- sparkdq.core.base_check (BaseRowCheck, BaseAggregateCheck)
- sparkdq.core.base_config (shared config logic)
- sparkdq.core.check_results and severity enumerations
e. Execution Engine
- sparkdq.engine.BatchDQEngine (entry point)
- sparkdq.engine.batch.check_runner – runs each Check on a DataFrame
- sparkdq.engine.batch.validation_result – collects per-check outcomes
- sparkdq.engine.validation_summary – aggregates a final report (success/failure, metrics)
f. Check Implementations
- Row-level checks: sparkdq.checks.row_level.* (null, numeric, date, timestamp, contained)
- Aggregate-level checks: sparkdq.checks.aggregate.* (count, schema)
g. Reporting / Results
- ValidationResult objects returned by the engine
- Summary methods for printing or exporting results to logs/metrics/alerts

##### Relationships and Data Flow
- Ingest DataFrame → Build CheckSet → Pass CheckSet + DataFrame into BatchDQEngine
- BatchDQEngine uses the Factory to create Check instances from configs
- CheckRunner executes row-level checks (filter, count violations) and aggregate checks (DataFrame.agg)
- Results from each Check → ValidationResult → collated in ValidationSummary
- ValidationSummary → returned to user code (Python shell, CI pipeline, orchestrator)

##### Architectural Patterns and Principles
- Separation of Concerns: Definition (configs) vs. Instantiation (factory) vs. Execution (engine) vs. Reporting
- Factory Pattern: Decouples config schema from concrete Check classes
- Plugin-style: New checks can be added by dropping modules under sparkdq.checks
- Declarative + Programmatic API: Same engine handles both Python-native and dictionary-based definitions
- Pure-PySpark: No non-Spark dependencies, enabling zero-overhead integration into any Spark environment

##### Technologies and Frameworks
- PySpark (3.5.x)
- Python 3.10+
- pytest for tests under tests/
- Sphinx for docs under docs/source
- GitHub Actions CI (ci.yaml), pre-commit, Dependabot

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% External Systems / Spark Cluster
    subgraph "External Systems / Spark" 
        ES1["Data Sources<br/>files, tables, streams"]:::external
        ES2["SparkCluster<br/>(SparkSession, Driver, Executors)"]:::external
    end

    %% Check Definition Layer
    subgraph "Check Definition Layer" 
        CDS1["CheckSet<br/>(sparkdq/management/check_set.py)"]:::frontend
        CDS2["Config Classes<br/>(sparkdq.checks.*_config)"]:::frontend
        CDS3["Declarative Configs<br/>(YAML/JSON)"]:::frontend
    end

    %% Factory Layer
    subgraph "Factory Layer"
        F1["Config Registry<br/>(check_config_registry.py)"]:::green
        F2["Check Factory<br/>(check_factory.py)"]:::green
    end

    %% Core Check Abstractions
    subgraph "Core Check Abstractions"
        C1["BaseRow/AggregateCheck<br/>(base_check.py)"]:::green
        C2["BaseConfig Logic<br/>(base_config.py)"]:::green
        C3["CheckResults Model<br/>(check_results.py)"]:::green
        C4["Severity Enum<br/>(severity.py)"]:::green
    end

    %% Execution Engine
    subgraph "Execution Engine"
        E1["BatchDQEngine<br/>(dq_engine.py)"]:::green
        E2["Check Runner<br/>(check_runner.py)"]:::green
        E3["ValidationResult<br/>(validation_result.py)"]:::green
        E4["ValidationSummary<br/>(validation_summary.py)"]:::green
    end

    %% Check Implementations
    subgraph "Check Implementations"
        R1["Row-level Checks<br/>(sparkdq/checks/row_level)"]:::orange
        R2["Aggregate Checks<br/>(sparkdq/checks/aggregate)"]:::orange
    end

    %% Reporting / Results Interfaces
    subgraph "Reporting / Results"
        RP["Reporting Interfaces<br/>(validation_result.py, validation_summary.py)"]:::orange
    end

    %% User / CI
    U["User / CI Pipeline"]:::external

    %% Connections
    ES1 -->|"DataFrame"| ES2
    ES2 -->|"SparkSession"| E1
    ES2 -->|"SparkSession"| CDS1
    CDS2 -->|"used by"| CDS1
    CDS3 -->|"loaded by"| CDS1
    CDS1 -->|"CheckConfig objects"| F1
    F1 -->|"registry lookup"| F2
    F2 -->|"CheckConfig objects"| E1
    F2 -->|"Instantiate Checks"| C1
    C2 -->|"config logic"| C1
    C4 -->|"defines severity"| C3
    E1 -->|"runs engine"| E2
    C1 -->|"run logic"| E2
    E2 -->|"per-check result"| E3
    C3 -->|"wrap outcomes"| E3
    E3 -->|"collected results"| E4
    E4 -->|"summary report"| RP
    RP -->|"output"| U

    %% Click Events
    click ES2 "https://github.com/sparkdq-community/sparkdq/blob/main/sparkdq/engine/base_engine.py"
    click CDS1 "https://github.com/sparkdq-community/sparkdq/blob/main/sparkdq/management/check_set.py"
    click F1 "https://github.com/sparkdq-community/sparkdq/blob/main/sparkdq/factory/check_config_registry.py"
    click F2 "https://github.com/sparkdq-community/sparkdq/blob/main/sparkdq/factory/check_factory.py"
    click C1 "https://github.com/sparkdq-community/sparkdq/blob/main/sparkdq/core/base_check.py"
    click C2 "https://github.com/sparkdq-community/sparkdq/blob/main/sparkdq/core/base_config.py"
    click C3 "https://github.com/sparkdq-community/sparkdq/blob/main/sparkdq/core/check_results.py"
    click C4 "https://github.com/sparkdq-community/sparkdq/blob/main/sparkdq/core/severity.py"
    click E1 "https://github.com/sparkdq-community/sparkdq/blob/main/sparkdq/engine/batch/dq_engine.py"
    click E2 "https://github.com/sparkdq-community/sparkdq/blob/main/sparkdq/engine/batch/check_runner.py"
    click E3 "https://github.com/sparkdq-community/sparkdq/blob/main/sparkdq/engine/batch/validation_result.py"
    click E4 "https://github.com/sparkdq-community/sparkdq/blob/main/sparkdq/engine/validation_summary.py"
    click R1 "https://github.com/sparkdq-community/sparkdq/tree/main/sparkdq/checks/row_level"
    click R2 "https://github.com/sparkdq-community/sparkdq/tree/main/sparkdq/checks/aggregate"
    click RP "https://github.com/sparkdq-community/sparkdq/blob/main/sparkdq/engine/batch/validation_result.py"
    click RP "https://github.com/sparkdq-community/sparkdq/blob/main/sparkdq/engine/validation_summary.py"

    %% Styles
    classDef external fill:#dddddd,stroke:#888888,color:#000000;
    classDef frontend fill:#bbdefb,stroke:#2196f3,color:#0d47a1;
    classDef green fill:#c8e6c9,stroke:#388e3c,color:#1b5e20;
    classDef orange fill:#ffe0b2,stroke:#fb8c00,color:#e65100;
```
