# great-expectations/great_expectations architecture

> Architecture diagram of the GitHub repository great-expectations/great_expectations, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-03). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/great-expectations/great_expectations
- Repository: https://github.com/great-expectations/great_expectations

## Overview

##### Project type and purpose
- This is a large-scale, open-source Python library (“GX Core”) for data quality: it provides a framework to define, execute, and document data validation rules (“Expectations”) against various data sources.
- It is not a monolithic web application but rather a modular SDK and CLI tool with heavy integration into cloud and on-prem data systems.
- Key use cases: programmatic definition of data quality tests, execution on batches from file systems, databases, cloud storage, and automatic generation of validation results documentation.

##### Top-level component breakdown
- Core modules:
- great_expectations/core – definitions for batches, expectations, metrics, serialization
- great_expectations/data_context – configuration management (“DataContext”) and stores
- great_expectations/datasource (including fluent) – connectors to pandas/Spark/SQL/GCS/S3/Azure/etc
- great_expectations/execution_engine – runtime engines for Pandas, Spark, SQLAlchemy
- great_expectations/expectations/metrics – implementations of metric providers
- great_expectations/expectations – “Expectation” rule definitions and validators
- great_expectations/validator – orchestration of metric computations and expectation checks
- great_expectations/render – rendering validation results into Data Docs or other formats
- Stores and persistence:
- Stores under data_context/store – where expectations, validation results, metrics, checkpoints get saved (filesystem, JSON, cloud, in-memory)
- CLI & Scripts:
- scripts/ and cli/ under contrib – command-line interface to scaffold projects, run tests, upload to Algolia, build docs, etc.

##### Key interactions and data flow
a. User script or CLI issues a command (e.g., `context = gx.get_context()`) →
b. DataContext loads configuration and initializes: datasources, stores, renderers →
c. A DataSource is called to fetch a Batch (via a DataConnector) →
d. Batch flows into an ExecutionEngine (Pandas/Spark/SQL) →
e. Validator requests one or more Metrics from the engine →
f. Expectations consume metrics and produce ValidationResults →
g. Results persist into a ValidationResultsStore →
h. Renderers pick up results and generate Data Docs (HTML/Markdown) or trigger actions (Slack, email, etc.).

##### Architectural patterns and principles
- Modular plugin architecture: DataSources, ExecutionEngines, Stores, Renderers are all backend-agnostic and swappable.
- Factory pattern: used for creating contexts, checkpoints, suites, and validators.
- “Fluent” builder APIs vs. YAML-driven config: dual approach for flexibility.
- Separation of concerns:
- Core definitions (batches, expectations, metrics)
- Runtime engines (execution)
- Persistence (stores)
- Presentation (renderers/data docs)

1. “User / CLI” – entry point
2. “DataContext” – loads config, orchestrates modules
3. “DataSource / DataConnector” – fetches data batches
4. “ExecutionEngine” – Pandas, Spark, SQL
5. “Metrics” – column/pair/table metric providers
6. “Expectations” – validation rules
7. “Validator” – coordinates metrics → expectations
8. “Stores” – expectations store, validation results store, checkpoint store
9. “Renderers / Data Docs” – static site generation, email, Slack, OpsGenie
10. “External Systems” – S3, Azure Blob, GCS, Snowflake, Redshift, Databricks, databases

## Diagram (Mermaid)

```mermaid
graph TB
    UserCLI["User / CLI\n(entrypoint)"]:::entry
    CLI2["CLI & Scripts\n(scaffold, tests,\ndocs build)"]:::entry
    DataContext["DataContext\n(config management,\nstores orchestration)"]:::core
    YAML["great_expectations.yml\n(YAML config)"]:::core
    Datasource["DataSource / DataConnector\n(fetches data batches)"]:::core
    ExecutionEngine["ExecutionEngine\n(Pandas/Spark/SQL)"]:::core
    Validator["Validator\n(coordinates metrics\nand expectations)"]:::core
    Metrics["Metrics\n(metric providers)"]:::core
    Expectations["Expectations\n(validation rules)"]:::core
    Stores["Stores\n(expectations, validation,\ncheckpoint)"]:::store
    Renderers["Renderers / Data Docs\n(generation, actions)"]:::renderer
    External["External Systems\n(S3, GCS, DBs,\nSnowflake, etc.)"]:::external

    UserCLI --> DataContext
    UserCLI -.-> CLI2
    CLI2 --> DataContext
    DataContext --> YAML
    DataContext <--> Stores
    DataContext --> Datasource
    Datasource --> External
    Datasource --> ExecutionEngine
    ExecutionEngine --> Validator
    Validator --> Metrics
    Metrics --> Expectations
    Expectations -->|"writes results"| Stores
    Stores -->|"reads results"| Renderers
    Renderers --> External

    classDef core fill:#AED6F1,stroke:#1F618D;
    classDef external fill:#ABEBC6,stroke:#1D8348;
    classDef store fill:#D5D8DC,stroke:#566573;
    classDef renderer fill:#F9E79F,stroke:#B7950B;
    classDef entry fill:#E8DAEF,stroke:#7D3C98;

    click CoreDefinitions "https://github.com/great-expectations/great_expectations/tree/develop/great_expectations/core"
    click DataContext "https://github.com/great-expectations/great_expectations/tree/develop/great_expectations/data_context/data_context"
    click Stores "https://github.com/great-expectations/great_expectations/tree/develop/great_expectations/data_context/store"
    click Datasource "https://github.com/great-expectations/great_expectations/tree/develop/great_expectations/datasource"
    click Datasource "https://github.com/great-expectations/great_expectations/tree/develop/great_expectations/datasource/fluent"
    click ExecutionEngine "https://github.com/great-expectations/great_expectations/tree/develop/great_expectations/execution_engine"
    click Metrics "https://github.com/great-expectations/great_expectations/tree/develop/great_expectations/expectations/metrics"
    click Expectations "https://github.com/great-expectations/great_expectations/tree/develop/great_expectations/expectations"
    click Validator "https://github.com/great-expectations/great_expectations/tree/develop/great_expectations/validator"
    click Renderers "https://github.com/great-expectations/great_expectations/tree/develop/great_expectations/render"
    click CLI2 "https://github.com/great-expectations/great_expectations/tree/develop/contrib/cli"
    click CLI2 "https://github.com/great-expectations/great_expectations/tree/develop/assets/scripts"
```
