# dieg0r01/databricks_framework_bootcamp architecture

> Architecture diagram of the GitHub repository dieg0r01/databricks_framework_bootcamp, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-06-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/dieg0r01/databricks_framework_bootcamp
- Repository: https://github.com/dieg0r01/databricks_framework_bootcamp

## Overview

1. **What this repo is**
   - This is a **metadata-driven PySpark ingestion framework**.
   - The core idea is: **JSON metadata describes each dataflow**, and the engine uses that metadata to read input data, validate it, branch it into OK/KO paths, and write outputs.

2. **Top-level execution boundary**
   - `main.py` is the CLI entrypoint.
   - It selects the execution **profile** (`local` or `databricks`) and either runs one named dataflow or all configured dataflows.
   - This is the main boundary between “user command” and “Spark pipeline execution”.

3. **Configuration and runtime profiles**
   - `conf/local.conf` and `conf/databricks.conf` define environment-specific settings.
   - The local profile is designed for `Spark local[*]` and file paths under `data/`.
   - The Databricks profile keeps the same logical flow but changes runtime wiring for that environment, which implies the code is intended to be portable across local and Databricks execution.

4. **Metadata as the control plane**
   - `metadata/dataflows.json` is the central orchestration file.
   - It likely defines which source to read, which schema to apply, what field-level rules to enforce, and where to write results.
   - `lib/metadata_loader.py` loads this metadata, and `schemas/metadata-schema.json` validates that the metadata itself is well-formed before execution.

5. **Input data and source layer**
   - Sample input lives under `data/input/events/person/sample.jsonl`.
   - `lib/sources.py` is the source abstraction layer that turns metadata-defined inputs into Spark-readable datasets.
   - The repository structure suggests a source-oriented design rather than hardcoding a single file path or format.

6. **Schema and field validation**
   - `schemas/person.json` defines the business data shape for the sample domain.
   - `lib/schema_validator.py` validates incoming records against the declared schema.
   - `lib/field_validators.py` applies per-field or business-rule checks beyond structural schema validation.
   - This creates two validation boundaries: **structural correctness** and **domain/business correctness**.

7. **Dataflow engine and branching**
   - `lib/dataflow_engine.py` is the orchestration layer that ties everything together.
   - It likely coordinates: load metadata → build Spark session → read source → validate schema/fields → split records into **OK** and **KO** branches → hand off to writers.
   - This is the most important component for an architecture diagram because it represents the actual pipeline control flow.

8. **Transformations and output**
   - `lib/transformations.py` contains the record-level transformations applied after ingest/validation.
   - `lib/writers.py` is the sink boundary; the README explicitly says outputs go to **Delta** sinks.
   - In local mode, outputs land under `data/output/`; in Databricks, the same logical sink is expected to be redirected to the platform’s storage/runtime.

9. **Spark session and observability**
   - `lib/spark_session.py` centralizes SparkSession creation so the rest of the code does not depend on environment-specific Spark bootstrapping.
   - `lib/tracing.py` adds execution tracing/logging, and there are integration tests dedicated to trace logs.
   - This indicates the system is designed to be debuggable and auditable at the dataflow level, not just functionally correct.

10. **Benchmarking and synthetic data generation**
   - `scripts/generate_faker_data.py` creates synthetic event data using Faker.
   - `scripts/run_benchmark.py` runs performance tests over different data volumes.
   - `docs/benchmark-report.md` suggests benchmark results are part of the repository’s deliverables, so performance is a first-class concern alongside correctness.

11. **Testing strategy**
   - Unit tests cover configuration, metadata loading, schema validation, field validation, and tracing.
   - Integration tests cover end-to-end dataflow execution, benchmark behavior, and the sample “prueba_acceso” case.
   - This maps cleanly onto the architecture: isolated components are tested separately, then the full Spark pipeline is verified as a system.

12. **Practical architecture summary**
   - The system is best drawn as:
     **CLI → config/profile selection → Spark session → metadata loader → source reader → schema/field validators → branching/transformation → Delta writers → tracing/logging**
   - The key architectural style is **metadata-driven ETL/ELT with validation and bifurcation**, portable across local and Databricks runtimes.

Main entry points: [`lib/dataflow_engine.py`](https://github.com/dieg0r01/databricks_framework_bootcamp/blob/main/lib/dataflow_engine.py) (Engine), [`tests`](https://github.com/dieg0r01/databricks_framework_bootcamp/tree/main/tests) (Tests), [`lib/metadata_loader.py`](https://github.com/dieg0r01/databricks_framework_bootcamp/blob/main/lib/metadata_loader.py) (Metadata loader), [`lib/schema_validator.py`](https://github.com/dieg0r01/databricks_framework_bootcamp/blob/main/lib/schema_validator.py) (Schema check)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_grp_entry["Entry & Runtime"]
  node_main_py["main.py<br/>CLI entrypoint<br/>[main.py]"]
  node_config["Config<br/>[config.py]"]
end

subgraph group_grp_core["Pipeline Core"]
  node_spark_session["Spark session<br/>session factory<br/>[spark_session.py]"]
  node_dataflow_engine{{"Engine<br/>orchestrator<br/>[dataflow_engine.py]"}}
  node_metadata_loader["Metadata loader<br/>control plane loader<br/>[metadata_loader.py]"]
  node_sources["Sources<br/>input adapter<br/>[sources.py]"]
  node_schema_validator["Schema check<br/>structural validator"]
  node_field_validators["Field rules<br/>business validator"]
  node_transformations["Transforms<br/>record transform<br/>[transformations.py]"]
  node_writers["Delta writers<br/>sink adapter<br/>[writers.py]"]
  node_tracing["Tracing<br/>observability<br/>[tracing.py]"]
end

subgraph group_grp_assets["Contracts & Sample Data"]
  node_dataflows_json["Dataflows<br/>metadata catalog<br/>[dataflows.json]"]
  node_metadata_schema["Meta schema<br/>json schema"]
  node_person_schema["Person schema<br/>domain schema<br/>[person.json]"]
  node_sample_input["Sample input<br/>source data<br/>[sample.jsonl]"]
  node_output_data[("Output data<br/>sink data")]
end

subgraph group_grp_ops["Ops & Verification"]
  node_faker_script["Faker gen<br/>data generator"]
  node_benchmark_script["Benchmark<br/>perf runner<br/>[run_benchmark.py]"]
  node_tests["Tests<br/>verification suite"]
end

node_main_py -->|"selects profile"| node_config
node_main_py -->|"starts run"| node_dataflow_engine
node_config -->|"configures"| node_spark_session
node_spark_session -->|"provides Spark"| node_dataflow_engine
node_dataflow_engine -->|"loads plan"| node_metadata_loader
node_metadata_loader -->|"reads"| node_dataflows_json
node_metadata_loader -->|"validates"| node_metadata_schema
node_dataflow_engine -->|"reads input"| node_sources
node_sources -->|"loads example"| node_sample_input
node_dataflow_engine -->|"checks shape"| node_schema_validator
node_schema_validator -->|"uses schema"| node_person_schema
node_dataflow_engine -->|"applies rules"| node_field_validators
node_dataflow_engine -->|"transforms"| node_transformations
node_transformations -->|"hands off"| node_writers
node_writers -->|"writes local"| node_output_data
node_dataflow_engine -->|"emits traces"| node_tracing
node_faker_script -->|"generates"| node_sample_input
node_benchmark_script -->|"benchmarks"| node_dataflow_engine
node_tests -->|"verifies"| node_metadata_loader
node_tests -->|"verifies"| node_schema_validator
node_tests -->|"verifies"| node_field_validators
node_tests -->|"verifies"| node_tracing
node_tests -->|"exercises"| node_dataflow_engine

click node_main_py "https://github.com/dieg0r01/databricks_framework_bootcamp/blob/main/main.py"
click node_config "https://github.com/dieg0r01/databricks_framework_bootcamp/blob/main/lib/config.py"
click node_spark_session "https://github.com/dieg0r01/databricks_framework_bootcamp/blob/main/lib/spark_session.py"
click node_dataflow_engine "https://github.com/dieg0r01/databricks_framework_bootcamp/blob/main/lib/dataflow_engine.py"
click node_metadata_loader "https://github.com/dieg0r01/databricks_framework_bootcamp/blob/main/lib/metadata_loader.py"
click node_sources "https://github.com/dieg0r01/databricks_framework_bootcamp/blob/main/lib/sources.py"
click node_schema_validator "https://github.com/dieg0r01/databricks_framework_bootcamp/blob/main/lib/schema_validator.py"
click node_field_validators "https://github.com/dieg0r01/databricks_framework_bootcamp/blob/main/lib/field_validators.py"
click node_transformations "https://github.com/dieg0r01/databricks_framework_bootcamp/blob/main/lib/transformations.py"
click node_writers "https://github.com/dieg0r01/databricks_framework_bootcamp/blob/main/lib/writers.py"
click node_tracing "https://github.com/dieg0r01/databricks_framework_bootcamp/blob/main/lib/tracing.py"
click node_dataflows_json "https://github.com/dieg0r01/databricks_framework_bootcamp/blob/main/metadata/dataflows.json"
click node_metadata_schema "https://github.com/dieg0r01/databricks_framework_bootcamp/blob/main/schemas/metadata-schema.json"
click node_person_schema "https://github.com/dieg0r01/databricks_framework_bootcamp/blob/main/schemas/person.json"
click node_sample_input "https://github.com/dieg0r01/databricks_framework_bootcamp/blob/main/data/input/events/person/sample.jsonl"
click node_output_data "https://github.com/dieg0r01/databricks_framework_bootcamp/tree/main/data/output"
click node_faker_script "https://github.com/dieg0r01/databricks_framework_bootcamp/blob/main/scripts/generate_faker_data.py"
click node_benchmark_script "https://github.com/dieg0r01/databricks_framework_bootcamp/blob/main/scripts/run_benchmark.py"
click node_tests "https://github.com/dieg0r01/databricks_framework_bootcamp/tree/main/tests"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_main_py,node_config toneBlue
class node_spark_session,node_dataflow_engine,node_metadata_loader,node_sources,node_schema_validator,node_field_validators,node_transformations,node_writers,node_tracing toneAmber
class node_dataflows_json,node_metadata_schema,node_person_schema,node_sample_input,node_output_data toneMint
class node_faker_script,node_benchmark_script,node_tests toneRose
```

## Components

### Entry & Runtime

CLI entrypoint and environment-specific configuration

- [main.py](https://github.com/dieg0r01/databricks_framework_bootcamp/blob/main/main.py): CLI entrypoint · Starts execution, selects profile, and runs one or all configured dataflows. · `main.py`
- [Config](https://github.com/dieg0r01/databricks_framework_bootcamp/blob/main/lib/config.py): runtime config · Loads local or Databricks settings and resolves environment-specific paths. · `lib/config.py`

### Pipeline Core

Metadata-driven Spark ingestion, validation, transformation, and writing

- [Spark session](https://github.com/dieg0r01/databricks_framework_bootcamp/blob/main/lib/spark_session.py): session factory · Centralized SparkSession creation for the selected runtime profile. · `lib/spark_session.py`
- [Engine](https://github.com/dieg0r01/databricks_framework_bootcamp/blob/main/lib/dataflow_engine.py): orchestrator · Coordinates the end-to-end ingestion flow from metadata to outputs. · `lib/dataflow_engine.py`
- [Metadata loader](https://github.com/dieg0r01/databricks_framework_bootcamp/blob/main/lib/metadata_loader.py): control plane loader · Reads dataflow definitions from JSON and prepares them for execution. · `lib/metadata_loader.py`
- [Sources](https://github.com/dieg0r01/databricks_framework_bootcamp/blob/main/lib/sources.py): input adapter · Turns metadata-defined inputs into Spark-readable datasets. · `lib/sources.py`
- [Schema check](https://github.com/dieg0r01/databricks_framework_bootcamp/blob/main/lib/schema_validator.py): structural validator · Validates incoming records against the declared business schema. · `lib/schema_validator.py`
- [Field rules](https://github.com/dieg0r01/databricks_framework_bootcamp/blob/main/lib/field_validators.py): business validator · Applies per-field and domain-specific validation rules. · `lib/field_validators.py`
- [Transforms](https://github.com/dieg0r01/databricks_framework_bootcamp/blob/main/lib/transformations.py): record transform · Applies record-level transformations after ingest and validation. · `lib/transformations.py`
- [Delta writers](https://github.com/dieg0r01/databricks_framework_bootcamp/blob/main/lib/writers.py): sink adapter · Persists OK and KO outputs to the configured sink targets. · `lib/writers.py`
- [Tracing](https://github.com/dieg0r01/databricks_framework_bootcamp/blob/main/lib/tracing.py): observability · Adds execution logging and traceability across the pipeline. · `lib/tracing.py`

### Contracts & Sample Data

Metadata, schemas, and representative input/output assets

- [Dataflows](https://github.com/dieg0r01/databricks_framework_bootcamp/blob/main/metadata/dataflows.json): metadata catalog · Central orchestration metadata describing each dataflow. · `metadata/dataflows.json`
- [Meta schema](https://github.com/dieg0r01/databricks_framework_bootcamp/blob/main/schemas/metadata-schema.json): json schema · Validates that metadata documents are well-formed before execution. · `schemas/metadata-schema.json`
- [Person schema](https://github.com/dieg0r01/databricks_framework_bootcamp/blob/main/schemas/person.json): domain schema · Business data shape for the sample person event flow. · `schemas/person.json`
- [Sample input](https://github.com/dieg0r01/databricks_framework_bootcamp/blob/main/data/input/events/person/sample.jsonl): source data · Representative JSONL event data used by the sample flow. · `data/input/events/person/sample.jsonl`
- [Output data](https://github.com/dieg0r01/databricks_framework_bootcamp/tree/main/data/output): sink data · Local output area for generated or validated results. · `data/output`

### Ops & Verification

Benchmarking, synthetic data generation, and tests

- [Faker gen](https://github.com/dieg0r01/databricks_framework_bootcamp/blob/main/scripts/generate_faker_data.py): data generator · Creates synthetic event data for experiments and benchmarks. · `scripts/generate_faker_data.py`
- [Benchmark](https://github.com/dieg0r01/databricks_framework_bootcamp/blob/main/scripts/run_benchmark.py): perf runner · Runs throughput tests over varying data volumes. · `scripts/run_benchmark.py`
- [Tests](https://github.com/dieg0r01/databricks_framework_bootcamp/tree/main/tests): verification suite · Covers config, metadata, validation, tracing, benchmarks, and end-to-end execution. · `tests`

## Connections

- main.py → Config: selects profile · CLI chooses local or Databricks runtime settings.
- main.py → Engine: starts run · Entrypoint dispatches one or all configured dataflows.
- Config → Spark session: configures · Runtime settings feed Spark session construction.
- Spark session → Engine: provides Spark · The engine executes against a prepared SparkSession.
- Engine → Metadata loader: loads plan · The engine resolves which flow to execute from metadata.
- Metadata loader → Dataflows: reads · Metadata loader consumes the central dataflow catalog.
- Metadata loader → Meta schema: validates · Metadata documents are checked against the metadata schema.
- Engine → Sources: reads input · Engine asks source adapters to materialize the configured dataset.
- Sources → Sample input: loads example · Sample flow reads from the provided JSONL event data.
- Engine → Schema check: checks shape · Records are validated structurally against the domain schema.
- Schema check → Person schema: uses schema · Structural validation is driven by the declared business schema.
- Engine → Field rules: applies rules · Domain checks split valid and invalid records.
- Engine → Transforms: transforms · Valid records are normalized or enriched before writeout.
- Transforms → Delta writers: hands off · Transformed output is passed to sink writers.
- Delta writers → Output data: writes local · Local profile persists results under the output directory.
- Engine → Tracing: emits traces · Execution events and diagnostics are recorded during the flow.
- Faker gen → Sample input: generates · Synthetic data generation feeds the input corpus used in testing and benchmarks.
- Benchmark → Engine: benchmarks · Benchmark runner measures end-to-end pipeline performance.
- Tests → Metadata loader: verifies · Unit tests cover metadata loading behavior.
- Tests → Schema check: verifies · Unit tests cover structural validation logic.
- Tests → Field rules: verifies · Unit tests cover business-rule validation logic.
- Tests → Tracing: verifies · Unit tests and integration tests assert trace output.
- Tests → Engine: exercises · Integration tests cover full dataflow execution and benchmark scenarios.