How javiersdr/churn-pipeline works
1. What this repo is
- A small analytics-engineering pipeline for the Kaggle Telco Customer Churn dataset.
- The core boundary is: local CSV data in, modeled analytics tables and tests out.
2. Primary runtime stack
- dbt-core + dbt-duckdb do the transformation work.
- DuckDB is the embedded analytical database where the seed and models are built.
- There is no separate external warehouse in this repo; everything is designed to run locally and reproducibly.
3. Raw data ingestion boundary
- The raw source lives at data/raw/WA_Fn-UseC_-Telco-Customer-Churn.csv.
- dbt loads the dataset as a seed via dbt_churn/seeds/telco_churn.csv, which becomes the first managed table in the pipeline.
- This is the handoff from “external file” to “modeled data platform.”
4. dbt model layers
- The repo follows the standard dbt layering pattern:
- Staging: models/staging/stg_telco__customers.sql
- Intermediate: models/intermediate/int_churn_features.sql
- Marts: models/marts/fct_churn_analysis.sql
- Each layer narrows the responsibility: cleanup and type normalization, then business logic, then analysis-ready output.
5. Staging layer role
- Staging is the closest thing to a source-alignment layer.
- It renames columns, normalizes values, and fixes types so downstream models work from a consistent schema.
6. Intermediate layer role
- This is where business logic is introduced.
- The README indicates churn-related feature creation, customer flags, and derived metrics such as TotalCharges from MonthlyCharges * tenure.
- This layer is the main transformation boundary between raw customer records and analytical features.
7. Mart layer role
- fct_churn_analysis is the final output table for reporting, dashboards, or machine learning.
- It represents the curated, analysis-ready contract of the pipeline.
8. Data quality subsystem
- Tests are split between:
- standard dbt schema tests in models/schema.yml and model-specific YAML
- custom SQL tests in dbt_churn/tests/, including check_charges.sql and check_phone_logic.sql
- The project emphasizes quality gates: the README says there are 10 tests total, with one configured as a warning because null TotalCharges is valid for new customers.
9. Orchestration boundary
- airflow/dags/churn_pipeline.py defines the scheduled pipeline in Apache Airflow.
- Airflow’s job here is orchestration, not transformation: it triggers dbt execution on a daily cadence and manages the run sequence.
- The Airflow container is separate from the dbt container, which keeps scheduling concerns isolated from modeling concerns.
10. CI/CD boundary
- .github/workflows/dbt_ci.yml runs dbt checks on push to main.
- This acts as a second execution path for the same modeling and test logic, independent of the Airflow schedule.
- In architecture terms, CI protects the model contract; Airflow operationalizes it.
11. Containerization and local execution
- Dockerfile, Dockerfile.airflow, and docker-compose.yml package the pipeline into repeatable services.
- Docker Compose ties together the dbt runtime and Airflow, making the repo self-contained for local development and demos.
- The container boundary matters because it is the main deployment unit for both orchestration and transformations.
12. Dependency/vendor boundary
- dbt_packages/dbt_utils is vendored into the repo, so the project carries its dbt utility macros and related tests locally.
- This means the repository includes both its own pipeline code and a dependency layer that expands dbt’s built-in testing and SQL helper capabilities.
13. How data flows end to end
- Raw CSV → dbt seed → staging model → intermediate feature model → mart model → schema/custom tests.
- In parallel, Airflow schedules this flow daily, and GitHub Actions validates it on code changes.
- The architecture is therefore a local, file-backed ELT pipeline with two control planes: orchestration and CI.
14. Important diagram boundaries
- Separate these boxes in an architecture drawing:
- raw file storage
- dbt transformation engine
- DuckDB execution layer
- Airflow scheduler
- CI workflow
- container/runtime environment
- The main dependency direction is one-way: orchestration and CI invoke dbt; dbt reads seeds and writes modeled tables; tests validate the results.
Main entry points: dbt_project.yml dbt project · stg_telco__customers.sql Staging · int_churn_features.sql Features · fct_churn_analysis.sql Churn fact
Components 16
Raw data
External CSV input boundary.
- Raw CSV · Original Telco churn dataset on disk.
data/raw/WA_Fn-UseC_-Telco-Customer-Churn.csv
dbt pipeline
Transformation layers and quality checks.
- Churn seed · Seeded copy of the raw dataset loaded into DuckDB.
dbt_churn/seeds/telco_churn.csv - Staging · Source-alignment layer that renames, normalizes, and types customer fields.
dbt_churn/models/staging/stg_telco__customers.sql - Features · Intermediate churn feature engineering and derived metrics.
dbt_churn/models/intermediate/int_churn_features.sql - Churn fact · Final analysis-ready contract for reporting and downstream use.
dbt_churn/models/marts/fct_churn_analysis.sql - Schema tests · Model schema and YAML-based validation rules.
dbt_churn/models/schema.yml - Custom tests · Project-specific SQL checks for charge and phone logic.
dbt_churn/tests/check_charges.sql - Test config · YAML configuration for custom SQL tests.
dbt_churn/tests/check_charges.yml - dbt project · dbt project root defining models, seeds, and runtime behavior.
dbt_churn/dbt_project.yml - Profiles · Local connection profile for DuckDB execution.
dbt_churn/profiles.yml
Operations
Scheduling and CI control planes.
- Airflow DAG · Daily scheduler that triggers the dbt pipeline.
airflow/dags/churn_pipeline.py - GitHub CI · Push-to-main validation workflow for dbt checks.
.github/workflows/dbt_ci.yml
Runtime
Containerized local execution environment.
- Compose · Local multi-service setup tying dbt and Airflow together.
docker-compose.yml - dbt image · Container definition for the dbt runtime.
Dockerfile - Airflow image · Container definition for the Airflow scheduler runtime.
Dockerfile.airflow
Vendored deps
Checked-in dbt package dependency layer.
- dbt-utils · Checked-in dependency providing dbt utility macros and generic tests.
dbt_churn/dbt_packages/dbt_utils
Connections 21
- Raw CSV → Churn seed: ingested as · The raw file is loaded into the managed dbt seed table.
- Churn seed → Staging: modeled by · Staging reads from the seed and aligns the schema.
- Staging → Features: feeds · Clean customer records become input to feature engineering.
- Features → Churn fact: curated into · Intermediate logic rolls up into the final mart.
- Staging → Schema tests: validated by · Schema tests check the staging contract.
- Features → Schema tests: validated by · Schema tests also enforce intermediate model expectations.
- Churn fact → Schema tests: validated by · The mart is checked by model schema tests.
- Churn fact → Custom tests: validated by · Custom SQL tests verify business rules like charges and phone logic.
- Test config → Custom tests: configures · YAML defines the custom SQL test cases.
- dbt project → Churn seed: includes · Project configuration includes seed execution.
- dbt project → Staging: includes · Project configuration includes the staging model.
- dbt project → Features: includes · Project configuration includes the intermediate model.
- dbt project → Churn fact: includes · Project configuration includes the mart model.
- Profiles → dbt project: connects · Profiles supply the DuckDB connection target used by dbt.
- Airflow DAG → dbt project: runs · Airflow orchestrates dbt execution on a schedule.
- GitHub CI → dbt project: runs · CI invokes dbt validation on changes.
- Compose → dbt image: builds · Compose can build or run the dbt service image.
- Compose → Airflow image: builds · Compose can build or run the Airflow service image.
- dbt image → dbt project: packages · The dbt container carries the project runtime.
- Airflow image → Airflow DAG: packages · The Airflow container carries the scheduler and DAG.
- dbt-utils → Schema tests: extends · Vendored dbt-utils macros support generic tests and SQL helpers.