How nike-inc/spark-expectations works
Project Type & Purpose
• This is a PySpark-based data quality framework that “expects” (i.e., validates) data against predefined rules while ensuring that data failing those checks is quarantined. In addition to filtering and quarantining bad records, it generates metrics and even proactively triggers observability alerts via notifications.
• The README and key directories (especially spark_expectations/*) reveal that it is primarily a full-stack (or end-to-end) data quality solution for Spark data pipelines.
Main Components
a. Data Ingestion & Source:
• How raw data is read (for example, the examples show reading CSV files and loading tables).
• External data sources (e.g., CSV files, Kafka topics, external databases) as they are referenced in the examples and jar libraries.
b. Core Data Quality Engine:
• Spark_expectations/core module which contains the main functionality such as:
• Expectation definitions and checks (in expectations.py).
• Context management and exception handling.
c. Notification System:
• Spark_expectations/notifications module as a component managing alerting.
• Within it, show the plugins for email, Slack, Teams, and Zoom, which deliver notifications based on configurable triggers.
• Observability sub-feature, which uses a custom Jinja template to format the alert, as detailed in the README.
d. Data Sinks:
• Incorporate the sinks module (spark_expectations/sinks) which is responsible for writing out “good” and “error” records.
• Plugins like the Kafka writer that forward the error or validated data records downstream.
e. Configuration & Plugin System:
• Spark_expectations/config and secrets modules.
• This part of the architecture manages the global configuration parameters, user-defined settings for notifications, and any sensitive parameters (for SMTP or other integrations).
f. Utility Layer & Test Framework:
• A utilities layer that contains modules for actions, reading, regulating flow, and UDFs for custom operations.
• Also, indicate the tests directory as an essential part of the architecture that verifies each component.
g. Deployment & External Integrations:
• The presence of Docker scripts in the examples, the jars directory (which includes dependencies for Kafka and Spark integration), and GitHub workflows show that the project also integrates with deployment pipelines and CI/CD systems.
• CI/CD as a separate block (depicting GitHub Actions for build, static analysis, etc.) which interacts with code quality and deployment aspects of the project.
Relationships and Interactions
• That after passing through the core data quality engine, validated data continues to the “target” sink (for downstream consumption) while error records are split to an “error table.”
• That upon completion, statistics (from the processed data) are gathered and passed into the observability module which then pushes alerts via the notifications module.
• The user-configurable settings (loaded from the configuration module) govern the behavior of validation, notifications, and error-handling.
• That testing and CI/CD components interact with the code base via GitHub workflows.
Important Architectural Concepts
• Decorator Pattern: Highlight that user functions are wrapped using the @se.with_expectations decorator, meaning that data quality validations are injected seamlessly into user data pipelines.
• Plugin Architecture: Emphasize how notifications and sinks are designed to be modular (with various plugins available as seen in their subdirectories) so that extensions or integrations can be added without changing the core.
• Observability & Alerting: Explicitly call out the observability workflow (from extraction of detailed statistics, table generation, to alerting) to facilitate proactive monitoring.
• Data Flow Control: Capture the rule-based segmentation of data into valid (for downstream) and invalid (for error analysis) records, implying a “flow controller” within the core logic.