How dieg0r01/databricks_framework_bootcamp works
1. What this repo is
- This is a metadata-driven PySpark ingestion framework.
- The core idea is: JSON metadata describes each dataflow, and the engine uses that metadata to read input data, validate it, branch it into OK/KO paths, and write outputs.
2. Top-level execution boundary
- main.py is the CLI entrypoint.
- It selects the execution profile (local or databricks) and either runs one named dataflow or all configured dataflows.
- This is the main boundary between “user command” and “Spark pipeline execution”.
3. Configuration and runtime profiles
- conf/local.conf and conf/databricks.conf define environment-specific settings.
- The local profile is designed for Spark local[*] and file paths under data/.
- The Databricks profile keeps the same logical flow but changes runtime wiring for that environment, which implies the code is intended to be portable across local and Databricks execution.
4. Metadata as the control plane
- metadata/dataflows.json is the central orchestration file.
- It likely defines which source to read, which schema to apply, what field-level rules to enforce, and where to write results.
- lib/metadata_loader.py loads this metadata, and schemas/metadata-schema.json validates that the metadata itself is well-formed before execution.
5. Input data and source layer
- Sample input lives under data/input/events/person/sample.jsonl.
- lib/sources.py is the source abstraction layer that turns metadata-defined inputs into Spark-readable datasets.
- The repository structure suggests a source-oriented design rather than hardcoding a single file path or format.
6. Schema and field validation
- schemas/person.json defines the business data shape for the sample domain.
- lib/schema_validator.py validates incoming records against the declared schema.
- lib/field_validators.py applies per-field or business-rule checks beyond structural schema validation.
- This creates two validation boundaries: structural correctness and domain/business correctness.
7. Dataflow engine and branching
- lib/dataflow_engine.py is the orchestration layer that ties everything together.
- It likely coordinates: load metadata → build Spark session → read source → validate schema/fields → split records into OK and KO branches → hand off to writers.
- This is the most important component for an architecture diagram because it represents the actual pipeline control flow.
8. Transformations and output
- lib/transformations.py contains the record-level transformations applied after ingest/validation.
- lib/writers.py is the sink boundary; the README explicitly says outputs go to Delta sinks.
- In local mode, outputs land under data/output/; in Databricks, the same logical sink is expected to be redirected to the platform’s storage/runtime.
9. Spark session and observability
- lib/spark_session.py centralizes SparkSession creation so the rest of the code does not depend on environment-specific Spark bootstrapping.
- lib/tracing.py adds execution tracing/logging, and there are integration tests dedicated to trace logs.
- This indicates the system is designed to be debuggable and auditable at the dataflow level, not just functionally correct.
10. Benchmarking and synthetic data generation
- scripts/generate_faker_data.py creates synthetic event data using Faker.
- scripts/run_benchmark.py runs performance tests over different data volumes.
- docs/benchmark-report.md suggests benchmark results are part of the repository’s deliverables, so performance is a first-class concern alongside correctness.
11. Testing strategy
- Unit tests cover configuration, metadata loading, schema validation, field validation, and tracing.
- Integration tests cover end-to-end dataflow execution, benchmark behavior, and the sample “prueba_acceso” case.
- This maps cleanly onto the architecture: isolated components are tested separately, then the full Spark pipeline is verified as a system.
12. Practical architecture summary
- The system is best drawn as:
CLI → config/profile selection → Spark session → metadata loader → source reader → schema/field validators → branching/transformation → Delta writers → tracing/logging
- The key architectural style is metadata-driven ETL/ELT with validation and bifurcation, portable across local and Databricks runtimes.
Main entry points: dataflow_engine.py Engine · tests Tests · metadata_loader.py Metadata loader · schema_validator.py Schema check
Components 19
Entry & Runtime
CLI entrypoint and environment-specific configuration
Pipeline Core
Metadata-driven Spark ingestion, validation, transformation, and writing
- Spark session · Centralized SparkSession creation for the selected runtime profile.
lib/spark_session.py - Engine · Coordinates the end-to-end ingestion flow from metadata to outputs.
lib/dataflow_engine.py - Metadata loader · Reads dataflow definitions from JSON and prepares them for execution.
lib/metadata_loader.py - Sources · Turns metadata-defined inputs into Spark-readable datasets.
lib/sources.py - Schema check · Validates incoming records against the declared business schema.
lib/schema_validator.py - Field rules · Applies per-field and domain-specific validation rules.
lib/field_validators.py - Transforms · Applies record-level transformations after ingest and validation.
lib/transformations.py - Delta writers · Persists OK and KO outputs to the configured sink targets.
lib/writers.py - Tracing · Adds execution logging and traceability across the pipeline.
lib/tracing.py
Contracts & Sample Data
Metadata, schemas, and representative input/output assets
- Dataflows · Central orchestration metadata describing each dataflow.
metadata/dataflows.json - Meta schema · Validates that metadata documents are well-formed before execution.
schemas/metadata-schema.json - Person schema · Business data shape for the sample person event flow.
schemas/person.json - Sample input · Representative JSONL event data used by the sample flow.
data/input/events/person/sample.jsonl - Output data · Local output area for generated or validated results.
data/output
Ops & Verification
Benchmarking, synthetic data generation, and tests
Connections 23
- main.py → Config: selects profile · CLI chooses local or Databricks runtime settings.
- main.py → Engine: starts run · Entrypoint dispatches one or all configured dataflows.
- Config → Spark session: configures · Runtime settings feed Spark session construction.
- Spark session → Engine: provides Spark · The engine executes against a prepared SparkSession.
- Engine → Metadata loader: loads plan · The engine resolves which flow to execute from metadata.
- Metadata loader → Dataflows: reads · Metadata loader consumes the central dataflow catalog.
- Metadata loader → Meta schema: validates · Metadata documents are checked against the metadata schema.
- Engine → Sources: reads input · Engine asks source adapters to materialize the configured dataset.
- Sources → Sample input: loads example · Sample flow reads from the provided JSONL event data.
- Engine → Schema check: checks shape · Records are validated structurally against the domain schema.
- Schema check → Person schema: uses schema · Structural validation is driven by the declared business schema.
- Engine → Field rules: applies rules · Domain checks split valid and invalid records.
- Engine → Transforms: transforms · Valid records are normalized or enriched before writeout.
- Transforms → Delta writers: hands off · Transformed output is passed to sink writers.
- Delta writers → Output data: writes local · Local profile persists results under the output directory.
- Engine → Tracing: emits traces · Execution events and diagnostics are recorded during the flow.
- Faker gen → Sample input: generates · Synthetic data generation feeds the input corpus used in testing and benchmarks.
- Benchmark → Engine: benchmarks · Benchmark runner measures end-to-end pipeline performance.
- Tests → Metadata loader: verifies · Unit tests cover metadata loading behavior.
- Tests → Schema check: verifies · Unit tests cover structural validation logic.
- Tests → Field rules: verifies · Unit tests cover business-rule validation logic.
- Tests → Tracing: verifies · Unit tests and integration tests assert trace output.
- Tests → Engine: exercises · Integration tests cover full dataflow execution and benchmark scenarios.