How googlecloudplatform/professional-services-data-validator works
Main Components
• CLI Interface & Entry Point: Show how the command-line commands (e.g. “validate”, “generate-table-partitions”, “query”) are parsed. This is primarily driven by the “data_validation/__main__.py” and “cli_tools.py” components.
• Core Engine (Data Validation Library): Illustrate the central “data_validation” package that includes modules for:
• Validation logic (e.g. “data_validation/data_validation.py”, “schema_validation.py”, “validation_builder.py”)
• SQL query generation and query builder (located in “data_validation/query_builder/…”)
• Configuration management (“config_manager.py”)
• Helpers for connection management, secret management, state management, and error/exception handling.
• Result Handlers: Include modules that format and output validation results, especially the BigQuery result handler (in “data_validation/result_handlers/…”).
• Calculated/Custom Fields Module: Represent how the tool applies calculated transformations to columns (e.g. “jellyfish_distance.py” and the custom calculated fields logic in query_builder).
• Testing & CI/CD: Indicate supporting infrastructure such as the “tests” directory (unit tests, system tests) and the CI pipeline scripts and Dockerfiles found in the “ci” directory and “cloudbuild.yaml”.
• Deployment & Samples: Illustrate the deployment artifacts (Terraform scripts in “terraform” and multiple integration samples in the “samples” directory) that exemplify how the CLI tool can be run as part of larger workflows (e.g., Cloud Functions, Cloud Run, Airflow).
Relationships and Data Flow
• Command Parsing and Routing: Show how CLI input is processed and routed to the appropriate validation sub-module (column, row, schema, custom query).
• Results Processing: From the query execution, results are channeled to result handlers (e.g., BigQuery result handler) or printed to stdout.
• Error Handling & Logging: Illustrate logging and exception management aspects as integral parts of each flow.
• Testing and CI/CD: Optionally, show a separate area that handles testing (unit tests, integration tests) and CI/CD pipelines that build and deploy containers or run tasks in distributed environments.
Architectural Patterns
• Modular and Layered Architecture: The separation between CLI interface, business logic (validations) and integration (data connectors, result handlers) underlines modular design.
• Configuration-Driven Behavior: Highlight how YAML/JSON configuration supports easy customizations and scaling.
• Extensibility and Third-Party Integration: Note the “third_party” directory containing custom extensions for Ibis, illustrating extensibility.
• Testability: Emphasize the presence of both unit and system tests which validate individual modules and the overall system behavior.
• "Validation Engine" divided into sub-modules: "Aggregation/Column Validation", "Row Validation", "Schema Validation", "Custom Query Validation"
• "SQL Query Builder & Calculated Fields" showing dependency on the Ibis framework.
• "Configuration Manager" which interacts with YAML/JSON config files.
• "Result Handlers" for routing output to stdout or BigQuery.
• Data flow from user commands to the appropriate processing module.
• Interaction between the query builder and external databases.
• The process whereby CI/CD pipelines build, test, and deploy the validation tool.
• "Testing" (encompassing tests/unit, tests/system)
• "Deployment Samples" (from the samples directory)
• "Infrastructure as Code" (Terraform)