How mridul98/dbt_unit_test_best_practice works
Main Components
• Docker Environment – Represent the Docker environment as a container group that includes:
• PostgreSQL Database Container (referred to as “db” in the README)
• Adminer Client Container for UI-based database management.
• DBT Projects – Clearly identify two separate DBT projects:
• dbt_unit_testing: The original implementation (with directories such as models/iot, macros, seeds, snapshots, tests, and analyses).
• dbt_unit_testing_refactored: The refactored version with a more modular structure (including subdirectories like staging, intermediate, and reporting under models/iot and additional macros under macros/iot).
• Data Schema Setup – Show the DDL (ddl.txt) as the starting point for setting up the database tables.
• Python Environment – Include the setup of the Python environment (requirements.txt and virtual environment) that supports DBT operations.
Relationships and Data Flow
• That the Docker Compose flow is used to initialize and run the PostgreSQL (and Adminer) containers.
• In the refactored DBT project, illustrate the internal processing stages:
• Staging models load raw data from the sources.
• Intermediate models process the data.
• Reporting models generate final metrics or insights.
• How the DBT macros (and tests) are used to support transformations and ensure unit testing of models throughout the data pipeline.
Architectural Patterns & Design Principles
• Separation of concerns:
• Docker handles environment orchestration.
• DBT projects manage transformation logic and testing.
• SQL (ddl.txt) defines schema setup.
• That the modular directory structure in the refactored DBT project suggests a layered approach (staging > intermediate > reporting), which is useful for improved maintainability and clarity.
• That test-driven development is at the core, as unit tests and tests directories indicate the integration of automated testing in data pipeline development.