How databricks-solutions/lakeflow_framework works
Lakeflow Framework is a metadata-driven framework for configuring and running Databricks Spark Declarative Pipelines across batch and streaming workloads. The principal workflow is: initialize a pipeline, resolve its bundle configuration and specifications, construct dataflows, then declare source-to-target processing. The runtime supports multiple source, flow, and target patterns, plus CDC snapshots and operational metadata. The README documents broader medallion-architecture and modelling-pattern support. Some runtime modules were not excerpted, so uncertain internal dispatch relationships are omitted rather than inferred.
Main entry points: dlt_pipeline_builder.py Pipeline builder · dataflow_spec_builder.py Spec builder · dataflow_spec.py Dataflow specifications · dataflow.py Dataflow runtime
Components 19
Pipeline orchestration
- Pipeline builder
src/lakeflow_framework/dlt_pipeline_builder.py - Bundle loader
src/lakeflow_framework/bundle_loader.py
Configuration and specs
- Pipeline configuration
src/lakeflow_framework/pipeline_config.py - Pipeline details
src/lakeflow_framework/pipeline_details.py - Config resolver
src/lakeflow_framework/config_resolver.py - Substitution manager
src/lakeflow_framework/substitution_manager.py - Spec builder
src/lakeflow_framework/dataflow_spec_builder/dataflow_spec_builder.py - Spec mapping
src/lakeflow_framework/dataflow_spec_builder/spec_mapper.py - Template processing
src/lakeflow_framework/dataflow_spec_builder/template_processor.py - Shared utilities
src/lakeflow_framework/utility.py
Dataflow execution
- Dataflow specifications
src/lakeflow_framework/dataflow/dataflow_spec.py - Dataflow runtime
src/lakeflow_framework/dataflow/dataflow.py - Flow patterns
src/lakeflow_framework/dataflow/flows - CDC processing
src/lakeflow_framework/dataflow/cdc.py - CDC snapshots
src/lakeflow_framework/dataflow/cdc_snapshot.py - Operational metadata
src/lakeflow_framework/dataflow/operational_metadata.py
Sources and targets
- Source patterns
src/lakeflow_framework/dataflow/sources - Target patterns
src/lakeflow_framework/dataflow/targets
Other components
- Databricks pipeline
Connections 24
- Databricks pipeline → Pipeline builder: initializes
- Pipeline builder → Bundle loader: loads bundle
src/lakeflow_framework/dlt_pipeline_builder.py - Pipeline builder → Config resolver: resolves config
src/lakeflow_framework/dlt_pipeline_builder.py - Pipeline builder → Spec builder: builds specs
src/lakeflow_framework/dlt_pipeline_builder.py - Pipeline builder → Dataflow runtime: creates dataflows
src/lakeflow_framework/dlt_pipeline_builder.py - Pipeline builder → Pipeline configuration: initializes state
src/lakeflow_framework/dlt_pipeline_builder.py - Pipeline configuration → Pipeline details: stores details
src/lakeflow_framework/pipeline_config.py - Pipeline configuration → Substitution manager: stores manager
src/lakeflow_framework/pipeline_config.py - Spec builder → Spec mapping: uses mapping
src/lakeflow_framework/dataflow_spec_builder/dataflow_spec_builder.py - Spec builder → Template processing: uses templates
src/lakeflow_framework/dataflow_spec_builder/dataflow_spec_builder.py - Spec builder → Dataflow specifications: constructs specs
src/lakeflow_framework/dataflow_spec_builder/dataflow_spec_builder.py - Spec mapping → Config resolver: resolves config
src/lakeflow_framework/dataflow_spec_builder/spec_mapper.py - Spec builder → Shared utilities: validates specs
src/lakeflow_framework/dataflow_spec_builder/dataflow_spec_builder.py - Dataflow specifications → CDC processing: uses CDC settings
src/lakeflow_framework/dataflow/dataflow_spec.py - Dataflow specifications → CDC snapshots: uses snapshot settings
src/lakeflow_framework/dataflow/dataflow_spec.py - Dataflow specifications → Flow patterns: uses flow definitions
src/lakeflow_framework/dataflow/dataflow_spec.py - Dataflow runtime → Source patterns: reads sources
- Dataflow runtime → Flow patterns: applies flows
- Dataflow runtime → Target patterns: writes targets
- Source patterns → Operational metadata: adds metadata
src/lakeflow_framework/dataflow/sources/base.py - Source patterns → Shared utilities: uses utilities
src/lakeflow_framework/dataflow/sources/base.py - CDC snapshots → Databricks pipeline: declares snapshot flow
src/lakeflow_framework/dataflow/cdc_snapshot.py - Substitution manager → Shared utilities: loads and merges
src/lakeflow_framework/substitution_manager.py - Config resolver → Shared utilities: uses utilities
src/lakeflow_framework/config_resolver.py