How great-expectations/great_expectations works
Project type and purpose
• This is a large-scale, open-source Python library (“GX Core”) for data quality: it provides a framework to define, execute, and document data validation rules (“Expectations”) against various data sources.
• It is not a monolithic web application but rather a modular SDK and CLI tool with heavy integration into cloud and on-prem data systems.
• Key use cases: programmatic definition of data quality tests, execution on batches from file systems, databases, cloud storage, and automatic generation of validation results documentation.
Top-level component breakdown
• Core modules:
• great_expectations/core – definitions for batches, expectations, metrics, serialization
• great_expectations/data_context – configuration management (“DataContext”) and stores
• great_expectations/datasource (including fluent) – connectors to pandas/Spark/SQL/GCS/S3/Azure/etc
• great_expectations/execution_engine – runtime engines for Pandas, Spark, SQLAlchemy
• great_expectations/expectations/metrics – implementations of metric providers
• great_expectations/expectations – “Expectation” rule definitions and validators
• great_expectations/validator – orchestration of metric computations and expectation checks
• great_expectations/render – rendering validation results into Data Docs or other formats
• Stores and persistence:
• Stores under data_context/store – where expectations, validation results, metrics, checkpoints get saved (filesystem, JSON, cloud, in-memory)
• CLI & Scripts:
• scripts/ and cli/ under contrib – command-line interface to scaffold projects, run tests, upload to Algolia, build docs, etc.
Key interactions and data flow
a. User script or CLI issues a command (e.g., context = gx.get_context()) →
b. DataContext loads configuration and initializes: datasources, stores, renderers →
c. A DataSource is called to fetch a Batch (via a DataConnector) →
d. Batch flows into an ExecutionEngine (Pandas/Spark/SQL) →
e. Validator requests one or more Metrics from the engine →
f. Expectations consume metrics and produce ValidationResults →
g. Results persist into a ValidationResultsStore →
h. Renderers pick up results and generate Data Docs (HTML/Markdown) or trigger actions (Slack, email, etc.).
Architectural patterns and principles
• Modular plugin architecture: DataSources, ExecutionEngines, Stores, Renderers are all backend-agnostic and swappable.
• Factory pattern: used for creating contexts, checkpoints, suites, and validators.
• “Fluent” builder APIs vs. YAML-driven config: dual approach for flexibility.
• Separation of concerns:
• Core definitions (batches, expectations, metrics)
• Runtime engines (execution)
• Persistence (stores)
• Presentation (renderers/data docs)
1. “User / CLI” – entry point
2. “DataContext” – loads config, orchestrates modules
3. “DataSource / DataConnector” – fetches data batches
4. “ExecutionEngine” – Pandas, Spark, SQL
5. “Metrics” – column/pair/table metric providers
6. “Expectations” – validation rules
7. “Validator” – coordinates metrics → expectations
8. “Stores” – expectations store, validation results store, checkpoint store
9. “Renderers / Data Docs” – static site generation, email, Slack, OpsGenie
10. “External Systems” – S3, Azure Blob, GCS, Snowflake, Redshift, Databricks, databases