pavan164-ml/metaforge

Purpose

Metaforge (dbxmetagen) is documented as a Databricks utility for bulk-generating Unity Catalog Delta-table and column descriptions plus PI classifications. It supports LLM-generated comments or PII/PHI/PCI tags, an optional human-review workflow, and controlled write-back as Unity Catalog DDL.

Repository Boundary

The supplied repository tree contains only [README.md](README.md). All implementation paths, deployment files, notebooks, UI code, and job definitions referenced in that document are absent from the provided tree. Therefore, the architecture below describes the documented intended system, not a code-verifiable implementation.

Entry Points and Control Plane

The documented primary entry point is a Streamlit “MetaForge” UI, which submits generation or update requests to Databricks Jobs API 2.1. Generation is described as invoking separate data-driven and knowledge-driven notebook stages; update runs consume reviewed Excel files. The UI is intentionally asynchronous: it returns a Databricks job-run link rather than waiting for completion.

Metadata Generation Pipeline

1. A target-table list identifies Unity Catalog tables to process.

2. The job reads live schema, tags, constraints, and optionally sampled table rows.

3. Columns are grouped into configurable schema-aware chunks before LLM calls.

4. Prompts request either table/column comments or column PI classifications.

5. LLM JSON is validated against strict Pydantic response schemas and retried on malformed or mismatched output.

6. Generated results are persisted as reviewable DDL and metadata-log records.

The documented output boundary is Unity Catalog: comments become COMMENT ON / ALTER TABLE ... COMMENT; classifications become SET TAGS DDL. Automatic application is intended to be gated by an apply_ddl setting, defaulting to false.

Knowledge-Enriched Path

An optional retrieval path queries Databricks Vector Search indexes representing wiki, KT/transcript, and Git sources. Source-specific descriptions are generated independently, then an LLM synthesizes them into a consolidated table description. The metadata log retains source-specific fields, creating an attribution boundary between retrieved knowledge and synthesized output.

Review and Update Workflow

Generated metadata is exported to Excel or tabular files in a Unity Catalog Volume, optionally exchanged through SharePoint. A human reviews and approves edits. The update job reads approved files, merges them into the metadata log, generates final DDL, and optionally applies it to Unity Catalog. This separates generation from governance-sensitive mutation.

Runtime, Infrastructure, and External Services

The documented runtime is Python on Databricks job clusters, deployed through Databricks Asset Bundles. Core platform dependencies are Unity Catalog, Delta Lake/Volumes, Databricks Jobs, a Databricks OpenAI-compatible serving endpoint (documented default: Meta-Llama-3.3-70B), and Databricks Vector Search. Optional integrations include MLflow for model logging and Microsoft Graph/SharePoint for review-file workflow. The document also describes Databricks secret usage for tokens, although it flags hardcoded credentials and widget-passed SharePoint secrets as security risks.

Architectural Risks Documented

The README identifies sequential LLM processing as the scale bottleneck, with no confirmed checkpoint/resume path. It also flags inadequate environment isolation, a potentially invalid duplicate-key job YAML definition, and credential handling concerns. These claims require verification once the referenced source files are available.