# nike-inc/spark-expectations architecture

> Architecture diagram of the GitHub repository nike-inc/spark-expectations, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/nike-inc/spark-expectations
- Repository: https://github.com/nike-inc/spark-expectations

## Overview

##### Project Type & Purpose
- This is a PySpark-based data quality framework that “expects” (i.e., validates) data against predefined rules while ensuring that data failing those checks is quarantined. In addition to filtering and quarantining bad records, it generates metrics and even proactively triggers observability alerts via notifications.
- The README and key directories (especially spark_expectations/*) reveal that it is primarily a full-stack (or end-to-end) data quality solution for Spark data pipelines.

##### Main Components
a. Data Ingestion & Source:
- How raw data is read (for example, the examples show reading CSV files and loading tables).
- External data sources (e.g., CSV files, Kafka topics, external databases) as they are referenced in the examples and jar libraries.

b. Core Data Quality Engine:
- Spark_expectations/core module which contains the main functionality such as:
- Expectation definitions and checks (in expectations.py).
- Context management and exception handling.

c. Notification System:
- Spark_expectations/notifications module as a component managing alerting.
- Within it, show the plugins for email, Slack, Teams, and Zoom, which deliver notifications based on configurable triggers.
- Observability sub-feature, which uses a custom Jinja template to format the alert, as detailed in the README.

d. Data Sinks:
- Incorporate the sinks module (spark_expectations/sinks) which is responsible for writing out “good” and “error” records.
- Plugins like the Kafka writer that forward the error or validated data records downstream.

e. Configuration & Plugin System:
- Spark_expectations/config and secrets modules.
- This part of the architecture manages the global configuration parameters, user-defined settings for notifications, and any sensitive parameters (for SMTP or other integrations).

f. Utility Layer & Test Framework:
- A utilities layer that contains modules for actions, reading, regulating flow, and UDFs for custom operations.
- Also, indicate the tests directory as an essential part of the architecture that verifies each component.

g. Deployment & External Integrations:
- The presence of Docker scripts in the examples, the jars directory (which includes dependencies for Kafka and Spark integration), and GitHub workflows show that the project also integrates with deployment pipelines and CI/CD systems.
- CI/CD as a separate block (depicting GitHub Actions for build, static analysis, etc.) which interacts with code quality and deployment aspects of the project.

##### Relationships and Interactions
- That after passing through the core data quality engine, validated data continues to the “target” sink (for downstream consumption) while error records are split to an “error table.”
- That upon completion, statistics (from the processed data) are gathered and passed into the observability module which then pushes alerts via the notifications module.
- The user-configurable settings (loaded from the configuration module) govern the behavior of validation, notifications, and error-handling.
- That testing and CI/CD components interact with the code base via GitHub workflows.

##### Important Architectural Concepts
- Decorator Pattern: Highlight that user functions are wrapped using the @se.with_expectations decorator, meaning that data quality validations are injected seamlessly into user data pipelines.
- Plugin Architecture: Emphasize how notifications and sinks are designed to be modular (with various plugins available as seen in their subdirectories) so that extensions or integrations can be added without changing the core.
- Observability & Alerting: Explicitly call out the observability workflow (from extraction of detailed statistics, table generation, to alerting) to facilitate proactive monitoring.
- Data Flow Control: Capture the rule-based segmentation of data into valid (for downstream) and invalid (for error analysis) records, implying a “flow controller” within the core logic.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data Ingestion Layer
    subgraph "Data Ingestion Layer"
        DI1["Read Dataset"]
        DI2["CSV Resources"]
    end

    %% Core Data Quality Engine
    subgraph "Core Data Quality Engine"
        CE1["Expectations"]
        CE2["Context"]
        CE3["Exceptions"]
    end

    %% Notification Module
    subgraph "Notification Module"
        N1["Notifications Init"]
        N2["Notification Plugins"]
        N3["Notification Push"]
    end

    %% Data Sinks
    subgraph "Data Sinks"
        DS1["Sinks Init"]
        DS2["Sink Plugins"]
        DS3["Sinks Utils"]
    end

    %% Configuration & Secrets
    subgraph "Configuration & Plugin System"
        C1["Configuration"]
        C2["Secrets"]
    end

    %% Utility Layer & Testing
    subgraph "Utilities & Test Framework"
        UT1["Utilities"]
        UT2["Tests"]
    end

    %% Deployment & External Integrations
    subgraph "Deployment & External Integrations"
        DE1["CI/CD Workflows"]
        DE2["Docker Scripts"]
        DE3["Jars"]
    end

    %% Connections: Data Flow
    DI1 -->|"reads"| CE1
    DI2 -->|"provides data"| CE1
    CE1 -->|"valid/error records"| DS1
    CE1 -->|"emits alerts"| N3

    %% Internal Core Engine connections (logical grouping)
    CE1 --> CE2
    CE1 --> CE3

    %% Configuration influences core and notifications
    C1 -->|"configures"| CE1
    C1 -->|"configures"| N3

    %% Utilities support core and sinks
    UT1 -->|"supports"| CE1
    UT1 -->|"supports"| DS1

    %% Testing interacts with core components and notifications
    UT2 -->|"tests"| CE1
    UT2 -->|"tests"| N3
    UT2 -->|"tests"| DS1

    %% Deployment interacts with code base and pipelines
    DE1 -->|"triggers"| UT2
    DE1 -->|"updates"| C1
    DE2 --> DE1
    DE3 --> DE1

    %% Notification Module internal connections
    N1 --> N2
    N2 --> N3

    %% Data Sinks internal connections
    DS1 --> DS2
    DS1 --> DS3

    %% Click Events
    click DI1 "https://github.com/nike-inc/spark-expectations/blob/main/spark_expectations/examples/read_example_dataset.py"
    click DI2 "https://github.com/nike-inc/spark-expectations/tree/main/spark_expectations/examples/resources"
    click CE1 "https://github.com/nike-inc/spark-expectations/blob/main/spark_expectations/core/expectations.py"
    click CE2 "https://github.com/nike-inc/spark-expectations/blob/main/spark_expectations/core/context.py"
    click CE3 "https://github.com/nike-inc/spark-expectations/blob/main/spark_expectations/core/exceptions.py"
    click N1 "https://github.com/nike-inc/spark-expectations/blob/main/spark_expectations/notifications/__init__.py"
    click N2 "https://github.com/nike-inc/spark-expectations/tree/main/spark_expectations/notifications/plugins/"
    click N3 "https://github.com/nike-inc/spark-expectations/blob/main/spark_expectations/notifications/push/alert.py"
    click DS1 "https://github.com/nike-inc/spark-expectations/blob/main/spark_expectations/sinks/__init__.py"
    click DS2 "https://github.com/nike-inc/spark-expectations/tree/main/spark_expectations/sinks/plugins/"
    click DS3 "https://github.com/nike-inc/spark-expectations/blob/main/spark_expectations/sinks/utils/collect_statistics.py"
    click C1 "https://github.com/nike-inc/spark-expectations/tree/main/spark_expectations/config/"
    click C2 "https://github.com/nike-inc/spark-expectations/blob/main/spark_expectations/secrets/__init__.py"
    click UT1 "https://github.com/nike-inc/spark-expectations/blob/main/spark_expectations/utils/actions.py"
    click UT2 "https://github.com/nike-inc/spark-expectations/tree/main/tests/"
    click DE1 "https://github.com/nike-inc/spark-expectations/tree/main/.github/workflows/"
    click DE2 "https://github.com/nike-inc/spark-expectations/tree/main/spark_expectations/examples/docker_scripts/"
    click DE3 "https://github.com/nike-inc/spark-expectations/tree/main/jars/"

    %% Styles
    classDef ingestion fill:#cce5ff,stroke:#004085,stroke-width:2px;
    classDef core fill:#d4edda,stroke:#155724,stroke-width:2px;
    classDef notify fill:#fff3cd,stroke:#856404,stroke-width:2px;
    classDef sinks fill:#f8d7da,stroke:#721c24,stroke-width:2px;
    classDef config fill:#d1ecf1,stroke:#0c5460,stroke-width:2px;
    classDef util fill:#e2e3e5,stroke:#383d41,stroke-width:2px;
    classDef deploy fill:#fefefe,stroke:#6c757d,stroke-width:2px;

    class DI1,DI2 ingestion;
    class CE1,CE2,CE3 core;
    class N1,N2,N3 notify;
    class DS1,DS2,DS3 sinks;
    class C1,C2 config;
    class UT1,UT2 util;
    class DE1,DE2,DE3 deploy;
```
