# googlecloudplatform/dataplexutils architecture

> Architecture diagram of the GitHub repository googlecloudplatform/dataplexutils, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-22). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/googlecloudplatform/dataplexutils
- Repository: https://github.com/googlecloudplatform/dataplexutils

## Overview

##### Project Type and Purpose
- This is a full‑stack, multi‑interface metadata generation tool (“Metadata Wizard”) leveraging LLMs to automatically describe BigQuery tables/columns.
- It provides three interfaces:
- A RESTful backend API service (Python + FastAPI)
- A React‑based frontend app
- A CLI wrapper around the Python package
- Core functionality is packaged as a Python library (`dataplexutils.metadata.wizard`) and orchestrates: data gathering from BigQuery/Dataplex, prompt construction, LLM invocation on Vertex AI, and metadata updates back to BigQuery.

##### Main Components
A. Client Interfaces
1. Frontend (React)
– Directory: `src/frontend/metadata-wizard-app`
– Serves UI for users to input Dataplex/BigQuery parameters and view results.
2. CLI
– Directory: `src/cli/metadata_wizard_cli`
– Shell script wrapper (`install_cli.sh`) installs the package and exposes `metadata-wizard` command.
3. Jupyter Notebooks (`notebooks/`)
– Demo notebook for interactive experimentation.
B. Backend Service
– Directory: `src/backend_apis`
– FastAPI (or Flask) service defined in `main.py`, containerized via `Dockerfile`.
– Shell scripts to build/deploy locally or to Cloud run (`build_local.sh`, `build_deploy_cloud.sh`).
C. Core Library / Python Package
– Directory: `src/package/dataplexutils/metadata/wizard.py`
– Contains metadata‑generation logic, prompt templates (`constants.toml`), and versioning.
– Installable via `pyproject.toml` and `setup.py`.
D. Testing
– Directory: `tests/`
– Unit tests (`wizard_tests.py`), integration tests, and shell scripts to exercise CLI and API.
E. External Systems
1. Google BigQuery (table/column schemas, sample data)
2. Google Cloud Storage (optional CSV or PDF docs)
3. Vertex AI (LLM endpoint)
4. Dataplex (profile, lineage, quality metrics)

##### Relationships and Data Flow
- Frontend → Backend API (HTTPS requests carrying user inputs)
- CLI → Backend API (HTTP or local function calls if `--debug`)
- Backend API → Core Library (invokes `wizard.generate_metadata()`)
- Core Library → BigQuery Data API (gathers schema, sample rows)
- Core Library → Dataplex API (fetches lineage/quality/profile)
- Core Library → GCS API (downloads CSV/PDF docs)
- Core Library → Vertex AI (sends prompts, receives completion)
- Core Library → BigQuery Data API (updates table/column descriptions)

##### Architectural Patterns and Principles
- Layered architecture:
- Presentation (Frontend, CLI, Notebook)
- Service (Backend API)
- Domain Logic (Metadata Wizard library)
- Infrastructure (Google Cloud services)
- Separation of concerns: each interface is decoupled from core logic.
- Containerization: backend service isolated in Docker.
- Extensibility: prompt templates in TOML, strategy flags in CLI.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Presentation Layer
    subgraph "Presentation Layer"
        direction TB
        FEApp["React Frontend App"]:::frontend
        FEBuild["Frontend Build Script"]:::script
        CLIApp["CLI Interface"]:::cli
        InstallCli["CLI Install Script"]:::script
        NB["Jupyter Notebook"]:::notebook
    end

    %% Service Layer
    subgraph "Service Layer"
        direction TB
        subgraph "Backend Service"
            direction TB
            BE["FastAPI Backend API"]:::backend
            DF["Dockerfile"]:::script
            BL["build_local.sh"]:::script
            BC["build_deploy_cloud.sh"]:::script
        end
    end

    %% Core Library Layer
    subgraph "Core Library Layer"
        direction TB
        Wizard["wizard.py"]:::core
        Constants["constants.toml"]:::core
        Version["version.py"]:::core
        Pyproj["pyproject.toml"]:::core
        SetupCLI["setup.py"]:::core
        BuildScript["build_install_package.sh"]:::script
        Manifest["MANIFEST.in"]:::script
    end

    %% Testing Suite
    subgraph "Testing Suite"
        direction TB
        Tests["tests/"]:::script
    end

    %% Infrastructure Layer
    subgraph "Infrastructure Layer"
        direction TB
        BQ[/"BigQuery"/]:::service
        DP[(Dataplex)]:::service
        GCS[(GCS)]:::service
        VA((Vertex AI)):::service
    end

    %% Data Flows
    FEApp -->|HTTPS Request| BE
    CLIApp -->|HTTP Request or Local Call| BE
    NB -->|Direct Library Call| Wizard
    BE -->|"invoke generate_metadata()"| Wizard
    Wizard -->|"GET schema"| BQ
    Wizard -->|"GET profile/lineage"| DP
    Wizard -->|"DOWNLOAD docs"| GCS
    Wizard -->|"POST prompt"| VA
    VA -->|"Completion Response"| Wizard
    Wizard -->|"PATCH description"| BQ

    %% Deployment Flow
    BE --> DF
    BE --> BL
    BE --> BC

    %% Styling
    classDef frontend fill:#D0E8FF,stroke:#333,stroke-width:1px
    classDef cli fill:#E0FFD8,stroke:#333,stroke-width:1px
    classDef notebook fill:#F0F0F0,stroke:#333,stroke-width:1px
    classDef backend fill:#FFE5B4,stroke:#333,stroke-width:1px
    classDef core fill:#FFFACD,stroke:#333,stroke-width:1px
    classDef service fill:#D8F0FF,stroke:#333,stroke-width:1px
    classDef script fill:#FFD700,stroke:#333,stroke-width:1px

    %% Click Events
    click FEApp "https://github.com/googlecloudplatform/dataplexutils/tree/main/src/frontend/metadata-wizard-app"
    click FEBuild "https://github.com/googlecloudplatform/dataplexutils/blob/main/src/frontend/metadata-wizard-app/build_local.sh"
    click CLIApp "https://github.com/googlecloudplatform/dataplexutils/tree/main/src/cli/metadata_wizard_cli"
    click InstallCli "https://github.com/googlecloudplatform/dataplexutils/blob/main/src/cli/install_cli.sh"
    click NB "https://github.com/googlecloudplatform/dataplexutils/blob/main/notebooks/metadata_wizard_demo.ipynb"
    click BE "https://github.com/googlecloudplatform/dataplexutils/blob/main/src/backend_apis/main.py"
    click DF "https://github.com/googlecloudplatform/dataplexutils/tree/main/src/backend_apis/Dockerfile"
    click BL "https://github.com/googlecloudplatform/dataplexutils/blob/main/src/backend_apis/build_local.sh"
    click BC "https://github.com/googlecloudplatform/dataplexutils/blob/main/src/backend_apis/build_deploy_cloud.sh"
    click Wizard "https://github.com/googlecloudplatform/dataplexutils/blob/main/src/package/dataplexutils/metadata/wizard.py"
    click Constants "https://github.com/googlecloudplatform/dataplexutils/blob/main/src/package/dataplexutils/metadata/constants.toml"
    click Version "https://github.com/googlecloudplatform/dataplexutils/blob/main/src/package/dataplexutils/metadata/version.py"
    click Pyproj "https://github.com/googlecloudplatform/dataplexutils/blob/main/src/package/pyproject.toml"
    click SetupCLI "https://github.com/googlecloudplatform/dataplexutils/blob/main/src/cli/setup.py"
    click BuildScript "https://github.com/googlecloudplatform/dataplexutils/blob/main/src/package/build_install_package.sh"
    click Manifest "https://github.com/googlecloudplatform/dataplexutils/blob/main/src/package/MANIFEST.in"
    click Tests "https://github.com/googlecloudplatform/dataplexutils/tree/main/tests/"
```
