How sparkdq-community/sparkdq works
Project Type and Purpose
• SparkDQ is an open-source, Spark-native data-quality validation library for PySpark. It provides a programmatic and declarative API for defining, executing, and reporting both row-level and aggregate-level checks directly inside Spark pipelines.
Main Components
a. Data Source / Spark Cluster
• External systems (files, tables, streams) feed DataFrames into Spark
• The SparkSession and cluster resources (driver, executors) that run the pipeline
b. Check Definition Layer
• sparkdq.management.CheckSet
• Configuration classes in sparkdq.checks.*_config (e.g. NullCheckConfig)
• Declarative configs (YAML/JSON dictionaries) loaded via CheckSet.add_checks_from_dicts
c. Factory Layer
• sparkdq.factory.check_config_registry: Registry of available check types
• sparkdq.factory.check_factory: Instantiates concrete Check objects from configs
d. Core Check Abstractions
• sparkdq.core.base_check (BaseRowCheck, BaseAggregateCheck)
• sparkdq.core.base_config (shared config logic)
• sparkdq.core.check_results and severity enumerations
e. Execution Engine
• sparkdq.engine.BatchDQEngine (entry point)
• sparkdq.engine.batch.check_runner – runs each Check on a DataFrame
• sparkdq.engine.batch.validation_result – collects per-check outcomes
• sparkdq.engine.validation_summary – aggregates a final report (success/failure, metrics)
f. Check Implementations
• Row-level checks: sparkdq.checks.row_level.* (null, numeric, date, timestamp, contained)
• Aggregate-level checks: sparkdq.checks.aggregate.* (count, schema)
g. Reporting / Results
• ValidationResult objects returned by the engine
• Summary methods for printing or exporting results to logs/metrics/alerts
Relationships and Data Flow
• Ingest DataFrame → Build CheckSet → Pass CheckSet + DataFrame into BatchDQEngine
• BatchDQEngine uses the Factory to create Check instances from configs
• CheckRunner executes row-level checks (filter, count violations) and aggregate checks (DataFrame.agg)
• Results from each Check → ValidationResult → collated in ValidationSummary
• ValidationSummary → returned to user code (Python shell, CI pipeline, orchestrator)
Architectural Patterns and Principles
• Separation of Concerns: Definition (configs) vs. Instantiation (factory) vs. Execution (engine) vs. Reporting
• Factory Pattern: Decouples config schema from concrete Check classes
• Plugin-style: New checks can be added by dropping modules under sparkdq.checks
• Declarative + Programmatic API: Same engine handles both Python-native and dictionary-based definitions
• Pure-PySpark: No non-Spark dependencies, enabling zero-overhead integration into any Spark environment
Technologies and Frameworks
• PySpark (3.5.x)
• Python 3.10+
• pytest for tests under tests/
• Sphinx for docs under docs/source
• GitHub Actions CI (ci.yaml), pre-commit, Dependabot