How burhanuddin-nahargarwala/elt-testing-practice-materials works
Project Type and Purpose
• This is not a classic front–end/back–end app but a training/reference toolkit of Databricks notebooks (.dbc) illustrating ELT testing strategies.
• Purpose: Provide hands-on code and real-world scenarios for unit testing, data-quality checks, schema validation, CDC testing, business-logic validation, and logging/error-handling in ELT pipelines.
File Structure
• All top‐level files are Databricks notebooks (e.g., CSV Parsing.dbc, JSON Testing in ETL‐ELT Pipelines.dbc, etc.).
• A README.md describes content and usage but no app code or deployment scripts.
README for Insights
• Confirms key testing categories (unit tests, data quality, schema changes, incremental loads, CDC, business rules).
• Emphasizes execution within a Databricks workspace.
Main Components
a. Data Sources (left side)
• Relational DBs / CSV exports
• JSON files
• Excel files / other raw feeds
b. Databricks Environment (center)
• Workspace / Cluster
• Notebook Modules – categorize by testing type
– Unit Testing Notebook(s)
– Data Quality & Null/Duplicate Checks Notebook(s)
– Schema Validation & Change‐Detection Notebook(s)
– Incremental Load & CDC Testing Notebook(s)
– Business Logic Validation Notebook(s)
– Logging & Error‐Handling Practices Notebook(s)
c. Supporting Services (right side)
• Test Data Lake / Silver / Gold tables
• Metrics & Logging Store (e.g., Delta tables or external monitoring)
• Dashboard / Reporting (optional)
Relationships & Data Flows
• Outputs from notebooks (e.g., test results, validated tables, error logs) flow out to the Test Data Lake or logging store.
• Optionally, a feedback loop back to developers/learners (e.g., notifications).
Architectural Patterns & Technologies
• Notebook‐centric modular testing: each .dbc file represents one facet of the testing strategy.
• Delta Lake as target storage for transformed/tested data sets.
• Spark engines on Databricks clusters.
• Python/SQL code within notebooks.
• CI/CD hooks if integrated (e.g., automated runs on cluster startup).