# fivetran/great_expectations architecture

> Architecture diagram of the GitHub repository fivetran/great_expectations, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/fivetran/great_expectations
- Repository: https://github.com/fivetran/great_expectations

## Overview

GX Core is a data-quality framework: teams define Expectations, connect data sources, load batches, and evaluate data against those expectations to produce validation results that can be shared as documentation. The README establishes those product capabilities; sampled code supports the batch-management and execution-engine layers, cloud datasource integrations, and query-comparison expectation behavior. The supplied tree is selective, so some larger orchestration and result-rendering components are represented only at a documented level, without inferred call links.

Main entry points: [`great_expectations/datasource/fluent/data_connector/azure_blob_storage_data_connector.py`](https://github.com/fivetran/great_expectations/blob/develop/great_expectations/datasource/fluent/data_connector/azure_blob_storage_data_connector.py) (Azure connector), [`great_expectations/execution_engine/execution_engine.py`](https://github.com/fivetran/great_expectations/blob/develop/great_expectations/execution_engine/execution_engine.py) (Execution engine), [`great_expectations/datasource/fluent/pandas_azure_blob_storage_datasource.py`](https://github.com/fivetran/great_expectations/blob/develop/great_expectations/datasource/fluent/pandas_azure_blob_storage_datasource.py) (Pandas Azure), [`great_expectations/datasource/fluent/spark_azure_blob_storage_datasource.py`](https://github.com/fivetran/great_expectations/blob/develop/great_expectations/datasource/fluent/spark_azure_blob_storage_datasource.py) (Spark Azure)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_sources["Data Sources"]
  node_pandasazure["Pandas Azure"]
  node_sparkazure["Spark Azure"]
  node_sparkgcs["Spark GCS"]
  node_azureconnector["Azure connector"]
  node_gcsconnector["GCS connector"]
end

subgraph group_runtime["Validation Runtime"]
  node_batches["Batch manager<br/>[batch_manager.py]"]
  node_engine["Execution engine"]
  node_sqlengine["SQL engine"]
  node_sparkengine["Spark engine"]
end

subgraph group_expectations["Quality Rules"]
  node_expectations["Expectations"]
  node_querycompare["Query comparison"]
end

subgraph group_outcomes["Validation Outcomes"]
  node_validation["Validation"]
  node_results["Validation results"]
  node_documentation["Result documentation"]
end

node_team(("Data team"))
node_data[("Data stores")]
node_azure[("Azure Blob Storage")]
node_gcs[("Google Cloud Storage")]

node_team -->|"configures source"| node_pandasazure
node_team -->|"configures source"| node_sparkazure
node_team -->|"configures source"| node_sparkgcs
node_pandasazure -->|"builds connector"| node_azureconnector
node_sparkazure -->|"builds connector"| node_azureconnector
node_sparkgcs -->|"builds connector"| node_gcsconnector
node_azureconnector -->|"discovers files"| node_azure
node_gcsconnector -->|"discovers files"| node_gcs
node_team -->|"defines rules"| node_expectations
node_team -->|"requests checks"| node_validation
node_batches -->|"manages batch data"| node_engine
node_sqlengine -->|"implements"| node_engine
node_sparkengine -->|"implements"| node_engine
node_querycompare -->|"checks query results"| node_validation
node_validation -->|"produces"| node_results
node_results -.->|"documents outcomes"| node_documentation

click node_pandasazure "https://github.com/fivetran/great_expectations/blob/develop/great_expectations/datasource/fluent/pandas_azure_blob_storage_datasource.py"
click node_sparkazure "https://github.com/fivetran/great_expectations/blob/develop/great_expectations/datasource/fluent/spark_azure_blob_storage_datasource.py"
click node_sparkgcs "https://github.com/fivetran/great_expectations/blob/develop/great_expectations/datasource/fluent/spark_google_cloud_storage_datasource.py"
click node_azureconnector "https://github.com/fivetran/great_expectations/blob/develop/great_expectations/datasource/fluent/data_connector/azure_blob_storage_data_connector.py"
click node_gcsconnector "https://github.com/fivetran/great_expectations/blob/develop/great_expectations/datasource/fluent/data_connector/google_cloud_storage_data_connector.py"
click node_batches "https://github.com/fivetran/great_expectations/blob/develop/great_expectations/core/batch_manager.py"
click node_engine "https://github.com/fivetran/great_expectations/blob/develop/great_expectations/execution_engine/execution_engine.py"
click node_sqlengine "https://github.com/fivetran/great_expectations/blob/develop/great_expectations/execution_engine/sqlalchemy_execution_engine.py"
click node_sparkengine "https://github.com/fivetran/great_expectations/blob/develop/great_expectations/execution_engine/sparkdf_execution_engine.py"
click node_expectations "https://github.com/fivetran/great_expectations/tree/develop/great_expectations/expectations/core"
click node_querycompare "https://github.com/fivetran/great_expectations/blob/develop/great_expectations/expectations/core/expect_query_results_to_match_comparison.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_pandasazure,node_sparkazure,node_sparkgcs,node_azureconnector,node_gcsconnector toneBlue
class node_batches,node_engine,node_sqlengine,node_sparkengine,node_data,node_azure,node_gcs toneAmber
class node_expectations,node_querycompare toneMint
class node_validation,node_results,node_documentation toneRose
class node_team toneIndigo
```

## Components

### Data Sources

- [Pandas Azure](https://github.com/fivetran/great_expectations/blob/develop/great_expectations/datasource/fluent/pandas_azure_blob_storage_datasource.py): `great_expectations/datasource/fluent/pandas_azure_blob_storage_datasource.py`
- [Spark Azure](https://github.com/fivetran/great_expectations/blob/develop/great_expectations/datasource/fluent/spark_azure_blob_storage_datasource.py): `great_expectations/datasource/fluent/spark_azure_blob_storage_datasource.py`
- [Spark GCS](https://github.com/fivetran/great_expectations/blob/develop/great_expectations/datasource/fluent/spark_google_cloud_storage_datasource.py): `great_expectations/datasource/fluent/spark_google_cloud_storage_datasource.py`
- [Azure connector](https://github.com/fivetran/great_expectations/blob/develop/great_expectations/datasource/fluent/data_connector/azure_blob_storage_data_connector.py): `great_expectations/datasource/fluent/data_connector/azure_blob_storage_data_connector.py`
- [GCS connector](https://github.com/fivetran/great_expectations/blob/develop/great_expectations/datasource/fluent/data_connector/google_cloud_storage_data_connector.py): `great_expectations/datasource/fluent/data_connector/google_cloud_storage_data_connector.py`

### Validation Runtime

- [Batch manager](https://github.com/fivetran/great_expectations/blob/develop/great_expectations/core/batch_manager.py): `great_expectations/core/batch_manager.py`
- [Execution engine](https://github.com/fivetran/great_expectations/blob/develop/great_expectations/execution_engine/execution_engine.py): `great_expectations/execution_engine/execution_engine.py`
- [SQL engine](https://github.com/fivetran/great_expectations/blob/develop/great_expectations/execution_engine/sqlalchemy_execution_engine.py): `great_expectations/execution_engine/sqlalchemy_execution_engine.py`
- [Spark engine](https://github.com/fivetran/great_expectations/blob/develop/great_expectations/execution_engine/sparkdf_execution_engine.py): `great_expectations/execution_engine/sparkdf_execution_engine.py`

### Quality Rules

- [Expectations](https://github.com/fivetran/great_expectations/tree/develop/great_expectations/expectations/core): `great_expectations/expectations/core`
- [Query comparison](https://github.com/fivetran/great_expectations/blob/develop/great_expectations/expectations/core/expect_query_results_to_match_comparison.py): `great_expectations/expectations/core/expect_query_results_to_match_comparison.py`

### Validation Outcomes

- **Validation**
- **Validation results**
- **Result documentation**

### Other components

- **Data team**
- **Data stores**
- **Azure Blob Storage**
- **Google Cloud Storage**

## Connections

- Data team → Pandas Azure: configures source
- Data team → Spark Azure: configures source
- Data team → Spark GCS: configures source
- Pandas Azure → Azure connector: builds connector
- Spark Azure → Azure connector: builds connector
- Spark GCS → GCS connector: builds connector
- Azure connector → Azure Blob Storage: discovers files
- GCS connector → Google Cloud Storage: discovers files
- Data team → Expectations: defines rules
- Data team → Validation: requests checks
- Batch manager → Execution engine: manages batch data
- SQL engine → Execution engine: implements
- Spark engine → Execution engine: implements
- Query comparison → Validation: checks query results
- Validation → Validation results: produces
- Validation results → Result documentation: documents outcomes