# databricks-solutions/lakeflow_framework architecture

> Architecture diagram of the GitHub repository databricks-solutions/lakeflow_framework, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-02). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/databricks-solutions/lakeflow_framework
- Repository: https://github.com/databricks-solutions/lakeflow_framework

## Overview

Lakeflow Framework is a metadata-driven framework for configuring and running Databricks Spark Declarative Pipelines across batch and streaming workloads. The principal workflow is: initialize a pipeline, resolve its bundle configuration and specifications, construct dataflows, then declare source-to-target processing. The runtime supports multiple source, flow, and target patterns, plus CDC snapshots and operational metadata. The README documents broader medallion-architecture and modelling-pattern support. Some runtime modules were not excerpted, so uncertain internal dispatch relationships are omitted rather than inferred.

Main entry points: [`src/lakeflow_framework/dlt_pipeline_builder.py`](https://github.com/databricks-solutions/lakeflow_framework/blob/main/src/lakeflow_framework/dlt_pipeline_builder.py) (Pipeline builder), [`src/lakeflow_framework/dataflow_spec_builder/dataflow_spec_builder.py`](https://github.com/databricks-solutions/lakeflow_framework/blob/main/src/lakeflow_framework/dataflow_spec_builder/dataflow_spec_builder.py) (Spec builder), [`src/lakeflow_framework/dataflow/dataflow_spec.py`](https://github.com/databricks-solutions/lakeflow_framework/blob/main/src/lakeflow_framework/dataflow/dataflow_spec.py) (Dataflow specifications), [`src/lakeflow_framework/dataflow/dataflow.py`](https://github.com/databricks-solutions/lakeflow_framework/blob/main/src/lakeflow_framework/dataflow/dataflow.py) (Dataflow runtime)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_orchestration["Pipeline orchestration"]
  node_pipeline_builder["Pipeline builder"]
  node_bundle_loader["Bundle loader<br/>[bundle_loader.py]"]
end

subgraph group_configuration["Configuration and specs"]
  node_pipeline_state["Pipeline configuration<br/>[pipeline_config.py]"]
  node_pipeline_details["Pipeline details"]
  node_config_resolver["Config resolver<br/>[config_resolver.py]"]
  node_substitutions["Substitution manager"]
  node_spec_builder["Spec builder"]
  node_spec_mapping["Spec mapping<br/>[spec_mapper.py]"]
  node_template_processing["Template processing"]
  node_utility["Shared utilities<br/>[utility.py]"]
end

subgraph group_dataflow["Dataflow execution"]
  node_dataflow_spec["Dataflow specifications<br/>[dataflow_spec.py]"]
  node_dataflow_runtime["Dataflow runtime<br/>[dataflow.py]"]
  node_flow_patterns["Flow patterns"]
  node_cdc["CDC processing<br/>[cdc.py]"]
  node_cdc_snapshot["CDC snapshots<br/>[cdc_snapshot.py]"]
  node_operational_metadata["Operational metadata"]
end

subgraph group_integrations["Sources and targets"]
  node_source_patterns["Source patterns"]
  node_target_patterns["Target patterns"]
end

node_databricks(("Databricks pipeline"))

node_databricks -.->|"initializes"| node_pipeline_builder
node_pipeline_builder -->|"loads bundle"| node_bundle_loader
node_pipeline_builder -->|"resolves config"| node_config_resolver
node_pipeline_builder -->|"builds specs"| node_spec_builder
node_pipeline_builder -->|"creates dataflows"| node_dataflow_runtime
node_pipeline_builder -.->|"initializes state"| node_pipeline_state
node_pipeline_state -->|"stores details"| node_pipeline_details
node_pipeline_state -->|"stores manager"| node_substitutions
node_spec_builder -->|"uses mapping"| node_spec_mapping
node_spec_builder -->|"uses templates"| node_template_processing
node_spec_builder -->|"constructs specs"| node_dataflow_spec
node_spec_mapping -->|"resolves config"| node_config_resolver
node_spec_builder -->|"validates specs"| node_utility
node_dataflow_spec -->|"uses CDC settings"| node_cdc
node_dataflow_spec -->|"uses snapshot settings"| node_cdc_snapshot
node_dataflow_spec -->|"uses flow definitions"| node_flow_patterns
node_dataflow_runtime -.->|"reads sources"| node_source_patterns
node_dataflow_runtime -.->|"applies flows"| node_flow_patterns
node_dataflow_runtime -.->|"writes targets"| node_target_patterns
node_source_patterns -->|"adds metadata"| node_operational_metadata
node_source_patterns -->|"uses utilities"| node_utility
node_cdc_snapshot -->|"declares snapshot flow"| node_databricks
node_substitutions -->|"loads and merges"| node_utility
node_config_resolver -->|"uses utilities"| node_utility

click node_pipeline_builder "https://github.com/databricks-solutions/lakeflow_framework/blob/main/src/lakeflow_framework/dlt_pipeline_builder.py"
click node_bundle_loader "https://github.com/databricks-solutions/lakeflow_framework/blob/main/src/lakeflow_framework/bundle_loader.py"
click node_pipeline_state "https://github.com/databricks-solutions/lakeflow_framework/blob/main/src/lakeflow_framework/pipeline_config.py"
click node_pipeline_details "https://github.com/databricks-solutions/lakeflow_framework/blob/main/src/lakeflow_framework/pipeline_details.py"
click node_config_resolver "https://github.com/databricks-solutions/lakeflow_framework/blob/main/src/lakeflow_framework/config_resolver.py"
click node_substitutions "https://github.com/databricks-solutions/lakeflow_framework/blob/main/src/lakeflow_framework/substitution_manager.py"
click node_spec_builder "https://github.com/databricks-solutions/lakeflow_framework/blob/main/src/lakeflow_framework/dataflow_spec_builder/dataflow_spec_builder.py"
click node_spec_mapping "https://github.com/databricks-solutions/lakeflow_framework/blob/main/src/lakeflow_framework/dataflow_spec_builder/spec_mapper.py"
click node_template_processing "https://github.com/databricks-solutions/lakeflow_framework/blob/main/src/lakeflow_framework/dataflow_spec_builder/template_processor.py"
click node_dataflow_spec "https://github.com/databricks-solutions/lakeflow_framework/blob/main/src/lakeflow_framework/dataflow/dataflow_spec.py"
click node_dataflow_runtime "https://github.com/databricks-solutions/lakeflow_framework/blob/main/src/lakeflow_framework/dataflow/dataflow.py"
click node_flow_patterns "https://github.com/databricks-solutions/lakeflow_framework/tree/main/src/lakeflow_framework/dataflow/flows"
click node_cdc "https://github.com/databricks-solutions/lakeflow_framework/blob/main/src/lakeflow_framework/dataflow/cdc.py"
click node_cdc_snapshot "https://github.com/databricks-solutions/lakeflow_framework/blob/main/src/lakeflow_framework/dataflow/cdc_snapshot.py"
click node_source_patterns "https://github.com/databricks-solutions/lakeflow_framework/tree/main/src/lakeflow_framework/dataflow/sources"
click node_target_patterns "https://github.com/databricks-solutions/lakeflow_framework/tree/main/src/lakeflow_framework/dataflow/targets"
click node_operational_metadata "https://github.com/databricks-solutions/lakeflow_framework/blob/main/src/lakeflow_framework/dataflow/operational_metadata.py"
click node_utility "https://github.com/databricks-solutions/lakeflow_framework/blob/main/src/lakeflow_framework/utility.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_pipeline_builder,node_bundle_loader toneBlue
class node_pipeline_state,node_pipeline_details,node_config_resolver,node_substitutions,node_spec_builder,node_spec_mapping,node_template_processing,node_utility toneAmber
class node_dataflow_spec,node_dataflow_runtime,node_flow_patterns,node_cdc,node_cdc_snapshot,node_operational_metadata toneMint
class node_source_patterns,node_target_patterns toneRose
class node_databricks toneIndigo
```

## Components

### Pipeline orchestration

- [Pipeline builder](https://github.com/databricks-solutions/lakeflow_framework/blob/main/src/lakeflow_framework/dlt_pipeline_builder.py): `src/lakeflow_framework/dlt_pipeline_builder.py`
- [Bundle loader](https://github.com/databricks-solutions/lakeflow_framework/blob/main/src/lakeflow_framework/bundle_loader.py): `src/lakeflow_framework/bundle_loader.py`

### Configuration and specs

- [Pipeline configuration](https://github.com/databricks-solutions/lakeflow_framework/blob/main/src/lakeflow_framework/pipeline_config.py): `src/lakeflow_framework/pipeline_config.py`
- [Pipeline details](https://github.com/databricks-solutions/lakeflow_framework/blob/main/src/lakeflow_framework/pipeline_details.py): `src/lakeflow_framework/pipeline_details.py`
- [Config resolver](https://github.com/databricks-solutions/lakeflow_framework/blob/main/src/lakeflow_framework/config_resolver.py): `src/lakeflow_framework/config_resolver.py`
- [Substitution manager](https://github.com/databricks-solutions/lakeflow_framework/blob/main/src/lakeflow_framework/substitution_manager.py): `src/lakeflow_framework/substitution_manager.py`
- [Spec builder](https://github.com/databricks-solutions/lakeflow_framework/blob/main/src/lakeflow_framework/dataflow_spec_builder/dataflow_spec_builder.py): `src/lakeflow_framework/dataflow_spec_builder/dataflow_spec_builder.py`
- [Spec mapping](https://github.com/databricks-solutions/lakeflow_framework/blob/main/src/lakeflow_framework/dataflow_spec_builder/spec_mapper.py): `src/lakeflow_framework/dataflow_spec_builder/spec_mapper.py`
- [Template processing](https://github.com/databricks-solutions/lakeflow_framework/blob/main/src/lakeflow_framework/dataflow_spec_builder/template_processor.py): `src/lakeflow_framework/dataflow_spec_builder/template_processor.py`
- [Shared utilities](https://github.com/databricks-solutions/lakeflow_framework/blob/main/src/lakeflow_framework/utility.py): `src/lakeflow_framework/utility.py`

### Dataflow execution

- [Dataflow specifications](https://github.com/databricks-solutions/lakeflow_framework/blob/main/src/lakeflow_framework/dataflow/dataflow_spec.py): `src/lakeflow_framework/dataflow/dataflow_spec.py`
- [Dataflow runtime](https://github.com/databricks-solutions/lakeflow_framework/blob/main/src/lakeflow_framework/dataflow/dataflow.py): `src/lakeflow_framework/dataflow/dataflow.py`
- [Flow patterns](https://github.com/databricks-solutions/lakeflow_framework/tree/main/src/lakeflow_framework/dataflow/flows): `src/lakeflow_framework/dataflow/flows`
- [CDC processing](https://github.com/databricks-solutions/lakeflow_framework/blob/main/src/lakeflow_framework/dataflow/cdc.py): `src/lakeflow_framework/dataflow/cdc.py`
- [CDC snapshots](https://github.com/databricks-solutions/lakeflow_framework/blob/main/src/lakeflow_framework/dataflow/cdc_snapshot.py): `src/lakeflow_framework/dataflow/cdc_snapshot.py`
- [Operational metadata](https://github.com/databricks-solutions/lakeflow_framework/blob/main/src/lakeflow_framework/dataflow/operational_metadata.py): `src/lakeflow_framework/dataflow/operational_metadata.py`

### Sources and targets

- [Source patterns](https://github.com/databricks-solutions/lakeflow_framework/tree/main/src/lakeflow_framework/dataflow/sources): `src/lakeflow_framework/dataflow/sources`
- [Target patterns](https://github.com/databricks-solutions/lakeflow_framework/tree/main/src/lakeflow_framework/dataflow/targets): `src/lakeflow_framework/dataflow/targets`

### Other components

- **Databricks pipeline**

## Connections

- Databricks pipeline → Pipeline builder: initializes
- Pipeline builder → Bundle loader: loads bundle (evidence: `src/lakeflow_framework/dlt_pipeline_builder.py`)
- Pipeline builder → Config resolver: resolves config (evidence: `src/lakeflow_framework/dlt_pipeline_builder.py`)
- Pipeline builder → Spec builder: builds specs (evidence: `src/lakeflow_framework/dlt_pipeline_builder.py`)
- Pipeline builder → Dataflow runtime: creates dataflows (evidence: `src/lakeflow_framework/dlt_pipeline_builder.py`)
- Pipeline builder → Pipeline configuration: initializes state (evidence: `src/lakeflow_framework/dlt_pipeline_builder.py`)
- Pipeline configuration → Pipeline details: stores details (evidence: `src/lakeflow_framework/pipeline_config.py`)
- Pipeline configuration → Substitution manager: stores manager (evidence: `src/lakeflow_framework/pipeline_config.py`)
- Spec builder → Spec mapping: uses mapping (evidence: `src/lakeflow_framework/dataflow_spec_builder/dataflow_spec_builder.py`)
- Spec builder → Template processing: uses templates (evidence: `src/lakeflow_framework/dataflow_spec_builder/dataflow_spec_builder.py`)
- Spec builder → Dataflow specifications: constructs specs (evidence: `src/lakeflow_framework/dataflow_spec_builder/dataflow_spec_builder.py`)
- Spec mapping → Config resolver: resolves config (evidence: `src/lakeflow_framework/dataflow_spec_builder/spec_mapper.py`)
- Spec builder → Shared utilities: validates specs (evidence: `src/lakeflow_framework/dataflow_spec_builder/dataflow_spec_builder.py`)
- Dataflow specifications → CDC processing: uses CDC settings (evidence: `src/lakeflow_framework/dataflow/dataflow_spec.py`)
- Dataflow specifications → CDC snapshots: uses snapshot settings (evidence: `src/lakeflow_framework/dataflow/dataflow_spec.py`)
- Dataflow specifications → Flow patterns: uses flow definitions (evidence: `src/lakeflow_framework/dataflow/dataflow_spec.py`)
- Dataflow runtime → Source patterns: reads sources
- Dataflow runtime → Flow patterns: applies flows
- Dataflow runtime → Target patterns: writes targets
- Source patterns → Operational metadata: adds metadata (evidence: `src/lakeflow_framework/dataflow/sources/base.py`)
- Source patterns → Shared utilities: uses utilities (evidence: `src/lakeflow_framework/dataflow/sources/base.py`)
- CDC snapshots → Databricks pipeline: declares snapshot flow (evidence: `src/lakeflow_framework/dataflow/cdc_snapshot.py`)
- Substitution manager → Shared utilities: loads and merges (evidence: `src/lakeflow_framework/substitution_manager.py`)
- Config resolver → Shared utilities: uses utilities (evidence: `src/lakeflow_framework/config_resolver.py`)
