How googlecloudplatform/dataplexutils works
Project Type and Purpose
• This is a full‑stack, multi‑interface metadata generation tool (“Metadata Wizard”) leveraging LLMs to automatically describe BigQuery tables/columns.
• It provides three interfaces:
• A RESTful backend API service (Python + FastAPI)
• A React‑based frontend app
• A CLI wrapper around the Python package
• Core functionality is packaged as a Python library (dataplexutils.metadata.wizard) and orchestrates: data gathering from BigQuery/Dataplex, prompt construction, LLM invocation on Vertex AI, and metadata updates back to BigQuery.
Main Components
A. Client Interfaces
1. Frontend (React)
– Directory: src/frontend/metadata-wizard-app
– Serves UI for users to input Dataplex/BigQuery parameters and view results.
2. CLI
– Directory: src/cli/metadata_wizard_cli
– Shell script wrapper (install_cli.sh) installs the package and exposes metadata-wizard command.
3. Jupyter Notebooks (notebooks/)
– Demo notebook for interactive experimentation.
B. Backend Service
– Directory: src/backend_apis
– FastAPI (or Flask) service defined in main.py, containerized via Dockerfile.
– Shell scripts to build/deploy locally or to Cloud run (build_local.sh, build_deploy_cloud.sh).
C. Core Library / Python Package
– Directory: src/package/dataplexutils/metadata/wizard.py
– Contains metadata‑generation logic, prompt templates (constants.toml), and versioning.
– Installable via pyproject.toml and setup.py.
D. Testing
– Directory: tests/
– Unit tests (wizard_tests.py), integration tests, and shell scripts to exercise CLI and API.
E. External Systems
1. Google BigQuery (table/column schemas, sample data)
2. Google Cloud Storage (optional CSV or PDF docs)
3. Vertex AI (LLM endpoint)
4. Dataplex (profile, lineage, quality metrics)
Relationships and Data Flow
• Frontend → Backend API (HTTPS requests carrying user inputs)
• CLI → Backend API (HTTP or local function calls if --debug)
• Backend API → Core Library (invokes wizard.generate_metadata())
• Core Library → BigQuery Data API (gathers schema, sample rows)
• Core Library → Dataplex API (fetches lineage/quality/profile)
• Core Library → GCS API (downloads CSV/PDF docs)
• Core Library → Vertex AI (sends prompts, receives completion)
• Core Library → BigQuery Data API (updates table/column descriptions)
Architectural Patterns and Principles
• Layered architecture:
• Presentation (Frontend, CLI, Notebook)
• Service (Backend API)
• Domain Logic (Metadata Wizard library)
• Infrastructure (Google Cloud services)
• Separation of concerns: each interface is decoupled from core logic.
• Containerization: backend service isolated in Docker.
• Extensibility: prompt templates in TOML, strategy flags in CLI.