# pavan164-ml/metaforge architecture

> Architecture diagram of the GitHub repository pavan164-ml/metaforge, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-08-02). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/pavan164-ml/metaforge
- Repository: https://github.com/pavan164-ml/metaforge

## Overview

#### Purpose
Metaforge (`dbxmetagen`) is documented as a Databricks utility for bulk-generating Unity Catalog Delta-table and column descriptions plus PI classifications. It supports LLM-generated comments or PII/PHI/PCI tags, an optional human-review workflow, and controlled write-back as Unity Catalog DDL.

#### Repository Boundary
The supplied repository tree contains only [README.md](README.md). All implementation paths, deployment files, notebooks, UI code, and job definitions referenced in that document are absent from the provided tree. Therefore, the architecture below describes the documented intended system, not a code-verifiable implementation.

#### Entry Points and Control Plane
The documented primary entry point is a Streamlit “MetaForge” UI, which submits generation or update requests to Databricks Jobs API 2.1. Generation is described as invoking separate data-driven and knowledge-driven notebook stages; update runs consume reviewed Excel files. The UI is intentionally asynchronous: it returns a Databricks job-run link rather than waiting for completion.

#### Metadata Generation Pipeline
1. A target-table list identifies Unity Catalog tables to process.
2. The job reads live schema, tags, constraints, and optionally sampled table rows.
3. Columns are grouped into configurable schema-aware chunks before LLM calls.
4. Prompts request either table/column comments or column PI classifications.
5. LLM JSON is validated against strict Pydantic response schemas and retried on malformed or mismatched output.
6. Generated results are persisted as reviewable DDL and metadata-log records.

The documented output boundary is Unity Catalog: comments become `COMMENT ON` / `ALTER TABLE ... COMMENT`; classifications become `SET TAGS` DDL. Automatic application is intended to be gated by an `apply_ddl` setting, defaulting to false.

#### Knowledge-Enriched Path
An optional retrieval path queries Databricks Vector Search indexes representing wiki, KT/transcript, and Git sources. Source-specific descriptions are generated independently, then an LLM synthesizes them into a consolidated table description. The metadata log retains source-specific fields, creating an attribution boundary between retrieved knowledge and synthesized output.

#### Review and Update Workflow
Generated metadata is exported to Excel or tabular files in a Unity Catalog Volume, optionally exchanged through SharePoint. A human reviews and approves edits. The update job reads approved files, merges them into the metadata log, generates final DDL, and optionally applies it to Unity Catalog. This separates generation from governance-sensitive mutation.

#### Runtime, Infrastructure, and External Services
The documented runtime is Python on Databricks job clusters, deployed through Databricks Asset Bundles. Core platform dependencies are Unity Catalog, Delta Lake/Volumes, Databricks Jobs, a Databricks OpenAI-compatible serving endpoint (documented default: Meta-Llama-3.3-70B), and Databricks Vector Search. Optional integrations include MLflow for model logging and Microsoft Graph/SharePoint for review-file workflow. The document also describes Databricks secret usage for tokens, although it flags hardcoded credentials and widget-passed SharePoint secrets as security risks.

#### Architectural Risks Documented
The README identifies sequential LLM processing as the scale bottleneck, with no confirmed checkpoint/resume path. It also flags inadequate environment isolation, a potentially invalid duplicate-key job YAML definition, and credential handling concerns. These claims require verification once the referenced source files are available.

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_control["Control Plane"]
  node_ui["MetaForge UI<br/>Streamlit app"]
  node_jobs{{"Jobs API<br/>Databricks Jobs 2.1"}}
end

subgraph group_generation["Generation Pipeline"]
  node_generation_job["Generation Job<br/>Databricks job"]
  node_targets["Target Tables<br/>table list"]
  node_catalog_reader["Catalog Reader<br/>metadata extraction"]
  node_chunker["Schema Chunker<br/>prompt preparation"]
  node_validator["Response Validation<br/>Pydantic schemas"]
  node_knowledge_synthesis["Knowledge Synthesis<br/>LLM enrichment"]
end

subgraph group_governance["Review &amp; Write-back"]
  node_metadata_log[("Metadata Log<br/>Delta records")]
  node_review_export["Review Files<br/>Excel or tabular files"]
  node_sharepoint["SharePoint<br/>optional Microsoft Graph integration"]
  node_human_review["Human Review<br/>approval workflow"]
  node_update_job["Update Job<br/>Databricks job"]
  node_ddl_gate{{"DDL Apply Gate<br/>apply_ddl control"}}
end

subgraph group_platform["Databricks Platform"]
  node_llm{{"LLM Serving Endpoint<br/>Databricks OpenAI-compatible"}}
  node_vector_search[("Vector Search")]
  node_unity_catalog[("Unity Catalog<br/>metadata target")]
  node_asset_bundles["Asset Bundles<br/>deployment"]
end

node_asset_bundles -->|"deploys"| node_generation_job
node_asset_bundles -->|"deploys"| node_update_job
node_ui -->|"submits run"| node_jobs
node_jobs -->|"starts"| node_generation_job
node_jobs -->|"starts"| node_update_job
node_targets -->|"selects"| node_catalog_reader
node_unity_catalog -->|"schema and samples"| node_catalog_reader
node_generation_job -->|"orchestrates"| node_catalog_reader
node_catalog_reader -->|"metadata context"| node_chunker
node_chunker -->|"generation prompts"| node_llm
node_llm -->|"JSON response"| node_validator
node_vector_search -->|"retrieved sources"| node_knowledge_synthesis
node_knowledge_synthesis -->|"synthesis prompts"| node_llm
node_validator -->|"validated metadata"| node_metadata_log
node_knowledge_synthesis -->|"attributed descriptions"| node_metadata_log
node_metadata_log -->|"exports"| node_review_export
node_review_export -.->|"optional exchange"| node_sharepoint
node_review_export -->|"review input"| node_human_review
node_sharepoint -.->|"optional delivery"| node_human_review
node_human_review -->|"approved edits"| node_update_job
node_update_job -->|"merges approvals"| node_metadata_log
node_update_job -->|"final DDL"| node_ddl_gate
node_ddl_gate -->|"COMMENT / SET TAGS"| node_unity_catalog

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_ui,node_jobs toneBlue
class node_generation_job,node_targets,node_catalog_reader,node_chunker,node_validator,node_knowledge_synthesis toneAmber
class node_metadata_log,node_review_export,node_sharepoint,node_human_review,node_update_job,node_ddl_gate toneMint
class node_llm,node_vector_search,node_unity_catalog,node_asset_bundles toneRose
```

## Components

### Control Plane

- **MetaForge UI**: Streamlit app · Submits asynchronous generation and update runs.
- **Jobs API**: Databricks Jobs 2.1 · Starts job runs and returns run links.

### Generation Pipeline

- **Generation Job**: Databricks job · Orchestrates data-driven and knowledge-driven metadata generation.
- **Target Tables**: table list · Identifies Unity Catalog tables to process.
- **Catalog Reader**: metadata extraction · Reads schemas, tags, constraints, and optional row samples.
- **Schema Chunker**: prompt preparation · Groups columns into configurable schema-aware chunks.
- **Response Validation**: Pydantic schemas · Validates LLM JSON and retries malformed or mismatched responses.
- **Knowledge Synthesis**: LLM enrichment · Produces source-specific descriptions and a consolidated result.

### Review & Write-back

- **Metadata Log**: Delta records · Stores generated metadata, source attribution, and review state.
- **Review Files**: Excel or tabular files · Reviewable metadata and DDL stored in a Unity Catalog Volume.
- **SharePoint**: optional Microsoft Graph integration · Optional exchange point for review files.
- **Human Review**: approval workflow · Approves and edits generated metadata before mutation.
- **Update Job**: Databricks job · Merges approved files and generates final DDL.
- **DDL Apply Gate**: apply_ddl control · Defaults to false before applying governed mutations.

### Databricks Platform

- **LLM Serving Endpoint**: Databricks OpenAI-compatible · Generates comments or PI classifications.
- **Vector Search**: Databricks Vector Search · Queries wiki, KT/transcript, and Git knowledge indexes.
- **Unity Catalog**: metadata target · Receives comments and classification tags through DDL.
- **Asset Bundles**: deployment · Deploys Python workloads to Databricks job clusters.

## Connections

- Asset Bundles → Generation Job: deploys
- Asset Bundles → Update Job: deploys
- MetaForge UI → Jobs API: submits run
- Jobs API → Generation Job: starts
- Jobs API → Update Job: starts
- Target Tables → Catalog Reader: selects
- Unity Catalog → Catalog Reader: schema and samples
- Generation Job → Catalog Reader: orchestrates
- Catalog Reader → Schema Chunker: metadata context
- Schema Chunker → LLM Serving Endpoint: generation prompts
- LLM Serving Endpoint → Response Validation: JSON response
- Vector Search → Knowledge Synthesis: retrieved sources
- Knowledge Synthesis → LLM Serving Endpoint: synthesis prompts
- Response Validation → Metadata Log: validated metadata
- Knowledge Synthesis → Metadata Log: attributed descriptions
- Metadata Log → Review Files: exports
- Review Files → SharePoint: optional exchange
- Review Files → Human Review: review input
- SharePoint → Human Review: optional delivery
- Human Review → Update Job: approved edits
- Update Job → Metadata Log: merges approvals
- Update Job → DDL Apply Gate: final DDL
- DDL Apply Gate → Unity Catalog: COMMENT / SET TAGS