# googlecloudplatform/professional-services-data-validator architecture

> Architecture diagram of the GitHub repository googlecloudplatform/professional-services-data-validator, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/googlecloudplatform/professional-services-data-validator
- Repository: https://github.com/googlecloudplatform/professional-services-data-validator

## Overview

##### Main Components
- CLI Interface & Entry Point: Show how the command-line commands (e.g. “validate”, “generate-table-partitions”, “query”) are parsed. This is primarily driven by the “data_validation/__main__.py” and “cli_tools.py” components.
- Core Engine (Data Validation Library): Illustrate the central “data_validation” package that includes modules for:
- Validation logic (e.g. “data_validation/data_validation.py”, “schema_validation.py”, “validation_builder.py”)
- SQL query generation and query builder (located in “data_validation/query_builder/…”)
- Configuration management (“config_manager.py”)
- Helpers for connection management, secret management, state management, and error/exception handling.
- Result Handlers: Include modules that format and output validation results, especially the BigQuery result handler (in “data_validation/result_handlers/…”).
- Calculated/Custom Fields Module: Represent how the tool applies calculated transformations to columns (e.g. “jellyfish_distance.py” and the custom calculated fields logic in query_builder).
- Testing & CI/CD: Indicate supporting infrastructure such as the “tests” directory (unit tests, system tests) and the CI pipeline scripts and Dockerfiles found in the “ci” directory and “cloudbuild.yaml”.
- Deployment & Samples: Illustrate the deployment artifacts (Terraform scripts in “terraform” and multiple integration samples in the “samples” directory) that exemplify how the CLI tool can be run as part of larger workflows (e.g., Cloud Functions, Cloud Run, Airflow).

##### Relationships and Data Flow
- Command Parsing and Routing: Show how CLI input is processed and routed to the appropriate validation sub-module (column, row, schema, custom query).
- Results Processing: From the query execution, results are channeled to result handlers (e.g., BigQuery result handler) or printed to stdout.
- Error Handling & Logging: Illustrate logging and exception management aspects as integral parts of each flow.
- Testing and CI/CD: Optionally, show a separate area that handles testing (unit tests, integration tests) and CI/CD pipelines that build and deploy containers or run tasks in distributed environments.

##### Architectural Patterns
- Modular and Layered Architecture: The separation between CLI interface, business logic (validations) and integration (data connectors, result handlers) underlines modular design.
- Configuration-Driven Behavior: Highlight how YAML/JSON configuration supports easy customizations and scaling.
- Extensibility and Third-Party Integration: Note the “third_party” directory containing custom extensions for Ibis, illustrating extensibility.
- Testability: Emphasize the presence of both unit and system tests which validate individual modules and the overall system behavior.

- "Validation Engine" divided into sub-modules: "Aggregation/Column Validation", "Row Validation", "Schema Validation", "Custom Query Validation"
- "SQL Query Builder & Calculated Fields" showing dependency on the Ibis framework.
- "Configuration Manager" which interacts with YAML/JSON config files.
- "Result Handlers" for routing output to stdout or BigQuery.
- Data flow from user commands to the appropriate processing module.
- Interaction between the query builder and external databases.
- The process whereby CI/CD pipelines build, test, and deploy the validation tool.
- "Testing" (encompassing tests/unit, tests/system)
- "Deployment Samples" (from the samples directory)
- "Infrastructure as Code" (Terraform)

## Diagram (Mermaid)

```mermaid
flowchart TD

    %% User Input
    U("User Command"):::input

    %% CLI Layer Subgraph
    subgraph "CLI Layer"
        CLI_Main["CLI: __main__.py"]:::cli
        CLI_Tools["CLI Tools: cli_tools.py"]:::cli
    end

    %% Core Validation Engine Subgraph
    subgraph "Core Validation Engine"
        VAL_Core["Validation Logic: data_validation.py"]:::engine
        Schema_Val["Schema Validation: schema_validation.py"]:::engine
        Val_Builder["Validation Builder: validation_builder.py"]:::engine
        LogErr["Logging & Error Handling"]:::error
    end

    %% Configuration Manager
    Config_Manager["Configuration Manager: config_manager.py"]:::config

    %% SQL Query Builder & Calculated Fields Subgraph
    subgraph "SQL Query Builder & Calculated Fields"
        SQL_Query_Builder["SQL Query Builder<br/>(query_builder/)"]:::query
        Calc_Fields["Calculated Fields:<br/>jellyfish_distance.py"]:::query
        Ibis_Integration["Ibis Integration<br/>(third_party/ibis/)"]:::external
    end

    %% External Data Sources Subgraph
    subgraph "External Data Sources"
        BigQuery["BigQuery"]:::database
        Oracle["Oracle"]:::database
        Postgres["Postgres"]:::database
    end

    %% Result Handlers
    Result_Handlers["Result Handlers<br/>(result_handlers/)"]:::result

    %% Testing & CI/CD
    Testing_CICD["Testing & CI/CD<br/>(tests/, ci/)"]:::testing

    %% Deployment & Samples
    Deployment_Samples["Deployment & Samples<br/>(samples/, terraform/)"]:::deployment

    %% Connections and Data Flow
    U -->|"invokes"| CLI_Main
    U -->|"invokes"| CLI_Tools

    CLI_Main -->|"parses commands"| VAL_Core
    CLI_Tools -->|"parses commands"| VAL_Core
    CLI_Main -->|"routes config"| Config_Manager
    CLI_Tools -->|"routes config"| Config_Manager

    Config_Manager -->|"loads configuration"| VAL_Core

    VAL_Core -->|"performs validation"| Schema_Val
    VAL_Core -->|"builds validation"| Val_Builder
    VAL_Core -->|"logs errors"| LogErr

    Val_Builder -->|"generates SQL"| SQL_Query_Builder
    Schema_Val -->|"generates SQL"| SQL_Query_Builder

    SQL_Query_Builder -->|"applies calculations"| Calc_Fields
    SQL_Query_Builder -->|"uses"| Ibis_Integration

    Ibis_Integration -->|"executes queries"| BigQuery
    Ibis_Integration -->|"executes queries"| Oracle
    Ibis_Integration -->|"executes queries"| Postgres

    BigQuery -->|"returns data"| Result_Handlers
    Oracle -->|"returns data"| Result_Handlers
    Postgres -->|"returns data"| Result_Handlers

    Result_Handlers -->|"formats & outputs"| U

    %% Auxiliary Connections for Testing and Deployment
    Testing_CICD ---|"validates"| VAL_Core
    Testing_CICD ---|"tests components"| SQL_Query_Builder
    Deployment_Samples ---|"deploys"| CLI_Main
    Deployment_Samples ---|"deploys"| SQL_Query_Builder

    %% Click Events
    click CLI_Main "https://github.com/googlecloudplatform/professional-services-data-validator/blob/develop/data_validation/__main__.py"
    click CLI_Tools "https://github.com/googlecloudplatform/professional-services-data-validator/blob/develop/data_validation/cli_tools.py"
    click VAL_Core "https://github.com/googlecloudplatform/professional-services-data-validator/blob/develop/data_validation/data_validation.py"
    click Schema_Val "https://github.com/googlecloudplatform/professional-services-data-validator/blob/develop/data_validation/schema_validation.py"
    click Val_Builder "https://github.com/googlecloudplatform/professional-services-data-validator/blob/develop/data_validation/validation_builder.py"
    click SQL_Query_Builder "https://github.com/googlecloudplatform/professional-services-data-validator/tree/develop/data_validation/query_builder/"
    click Calc_Fields "https://github.com/googlecloudplatform/professional-services-data-validator/blob/develop/data_validation/jellyfish_distance.py"
    click Config_Manager "https://github.com/googlecloudplatform/professional-services-data-validator/blob/develop/data_validation/config_manager.py"
    click Result_Handlers "https://github.com/googlecloudplatform/professional-services-data-validator/tree/develop/data_validation/result_handlers/"
    click Ibis_Integration "https://github.com/googlecloudplatform/professional-services-data-validator/tree/develop/third_party/ibis/"
    click Testing_CICD "https://github.com/googlecloudplatform/professional-services-data-validator/tree/develop/tests/"
    click Deployment_Samples "https://github.com/googlecloudplatform/professional-services-data-validator/tree/develop/samples/"

    %% Styles
    classDef input fill:#fff2cc,stroke:#b58900,stroke-width:2px;
    classDef cli fill:#ffcccc,stroke:#d64541,stroke-width:2px;
    classDef engine fill:#ccffcc,stroke:#28a745,stroke-width:2px;
    classDef config fill:#ffffcc,stroke:#f0c674,stroke-width:2px;
    classDef query fill:#ccccff,stroke:#6a51a3,stroke-width:2px;
    classDef external fill:#e6e6e6,stroke:#636363,stroke-width:2px;
    classDef database fill:#f4cccc,stroke:#cc0000,stroke-width:2px,stroke-dasharray: 5 5;
    classDef result fill:#ffe6cc,stroke:#e67e22,stroke-width:2px;
    classDef testing fill:#ffebcc,stroke:#f39c12,stroke-width:2px;
    classDef deployment fill:#e6f7ff,stroke:#3498db,stroke-width:2px;
    classDef error fill:#f0e68c,stroke:#b8860b,stroke-width:2px;
```
