# javiersdr/churn-pipeline architecture

> Architecture diagram of the GitHub repository javiersdr/churn-pipeline, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-05-20). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/javiersdr/churn-pipeline
- Repository: https://github.com/javiersdr/churn-pipeline

## Overview

1. **What this repo is**
   - A small analytics-engineering pipeline for the Kaggle Telco Customer Churn dataset.
   - The core boundary is: local CSV data in, modeled analytics tables and tests out.

2. **Primary runtime stack**
   - **dbt-core + dbt-duckdb** do the transformation work.
   - **DuckDB** is the embedded analytical database where the seed and models are built.
   - There is no separate external warehouse in this repo; everything is designed to run locally and reproducibly.

3. **Raw data ingestion boundary**
   - The raw source lives at `data/raw/WA_Fn-UseC_-Telco-Customer-Churn.csv`.
   - dbt loads the dataset as a **seed** via `dbt_churn/seeds/telco_churn.csv`, which becomes the first managed table in the pipeline.
   - This is the handoff from “external file” to “modeled data platform.”

4. **dbt model layers**
   - The repo follows the standard dbt layering pattern:
     - **Staging**: `models/staging/stg_telco__customers.sql`
     - **Intermediate**: `models/intermediate/int_churn_features.sql`
     - **Marts**: `models/marts/fct_churn_analysis.sql`
   - Each layer narrows the responsibility: cleanup and type normalization, then business logic, then analysis-ready output.

5. **Staging layer role**
   - Staging is the closest thing to a source-alignment layer.
   - It renames columns, normalizes values, and fixes types so downstream models work from a consistent schema.

6. **Intermediate layer role**
   - This is where business logic is introduced.
   - The README indicates churn-related feature creation, customer flags, and derived metrics such as `TotalCharges` from `MonthlyCharges * tenure`.
   - This layer is the main transformation boundary between raw customer records and analytical features.

7. **Mart layer role**
   - `fct_churn_analysis` is the final output table for reporting, dashboards, or machine learning.
   - It represents the curated, analysis-ready contract of the pipeline.

8. **Data quality subsystem**
   - Tests are split between:
     - standard dbt schema tests in `models/schema.yml` and model-specific YAML
     - custom SQL tests in `dbt_churn/tests/`, including `check_charges.sql` and `check_phone_logic.sql`
   - The project emphasizes quality gates: the README says there are 10 tests total, with one configured as a warning because null `TotalCharges` is valid for new customers.

9. **Orchestration boundary**
   - `airflow/dags/churn_pipeline.py` defines the scheduled pipeline in Apache Airflow.
   - Airflow’s job here is orchestration, not transformation: it triggers dbt execution on a daily cadence and manages the run sequence.
   - The Airflow container is separate from the dbt container, which keeps scheduling concerns isolated from modeling concerns.

10. **CI/CD boundary**
   - `.github/workflows/dbt_ci.yml` runs dbt checks on push to `main`.
   - This acts as a second execution path for the same modeling and test logic, independent of the Airflow schedule.
   - In architecture terms, CI protects the model contract; Airflow operationalizes it.

11. **Containerization and local execution**
   - `Dockerfile`, `Dockerfile.airflow`, and `docker-compose.yml` package the pipeline into repeatable services.
   - Docker Compose ties together the dbt runtime and Airflow, making the repo self-contained for local development and demos.
   - The container boundary matters because it is the main deployment unit for both orchestration and transformations.

12. **Dependency/vendor boundary**
   - `dbt_packages/dbt_utils` is vendored into the repo, so the project carries its dbt utility macros and related tests locally.
   - This means the repository includes both its own pipeline code and a dependency layer that expands dbt’s built-in testing and SQL helper capabilities.

13. **How data flows end to end**
   - Raw CSV → dbt seed → staging model → intermediate feature model → mart model → schema/custom tests.
   - In parallel, Airflow schedules this flow daily, and GitHub Actions validates it on code changes.
   - The architecture is therefore a local, file-backed ELT pipeline with two control planes: orchestration and CI.

14. **Important diagram boundaries**
   - Separate these boxes in an architecture drawing:
     - raw file storage
     - dbt transformation engine
     - DuckDB execution layer
     - Airflow scheduler
     - CI workflow
     - container/runtime environment
   - The main dependency direction is one-way: orchestration and CI invoke dbt; dbt reads seeds and writes modeled tables; tests validate the results.

Main entry points: [`dbt_churn/dbt_project.yml`](https://github.com/javiersdr/churn-pipeline/blob/main/dbt_churn/dbt_project.yml) (dbt project), [`dbt_churn/models/staging/stg_telco__customers.sql`](https://github.com/javiersdr/churn-pipeline/blob/main/dbt_churn/models/staging/stg_telco__customers.sql) (Staging), [`dbt_churn/models/intermediate/int_churn_features.sql`](https://github.com/javiersdr/churn-pipeline/blob/main/dbt_churn/models/intermediate/int_churn_features.sql) (Features), [`dbt_churn/models/marts/fct_churn_analysis.sql`](https://github.com/javiersdr/churn-pipeline/blob/main/dbt_churn/models/marts/fct_churn_analysis.sql) (Churn fact)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_raw["Raw data"]
  node_data_raw_csv["Raw CSV<br/>source file"]
end

subgraph group_dbt["dbt pipeline"]
  node_seed_telco_churn["Churn seed<br/>dbt seed<br/>[telco_churn.csv]"]
  node_stg_customers["Staging<br/>dbt model"]
  node_int_features["Features<br/>dbt model"]
  node_fct_analysis["Churn fact<br/>dbt mart"]
  node_schema_tests["Schema tests<br/>dbt tests<br/>[schema.yml]"]
  node_custom_tests["Custom tests<br/>sql tests<br/>[check_charges.sql]"]
  node_custom_tests_yaml["Test config<br/>[check_charges.yml]"]
  node_dbt_project["dbt project<br/>dbt config<br/>[dbt_project.yml]"]
  node_profiles["Profiles<br/>dbt config<br/>[profiles.yml]"]
end

subgraph group_ops["Operations"]
  node_airflow_dag["Airflow DAG<br/>orchestrator<br/>[churn_pipeline.py]"]
  node_github_ci["GitHub CI<br/>workflow<br/>[dbt_ci.yml]"]
end

subgraph group_runtime["Runtime"]
  node_compose["Compose<br/>runtime manifest<br/>[docker-compose.yml]"]
  node_dbt_dockerfile["dbt image<br/>container build"]
  node_airflow_dockerfile["Airflow image<br/>container build<br/>[Dockerfile.airflow]"]
end

subgraph group_deps["Vendored deps"]
  node_dbt_utils_pkg["dbt-utils<br/>vendored package"]
end

node_data_raw_csv -->|"ingested as"| node_seed_telco_churn
node_seed_telco_churn -->|"modeled by"| node_stg_customers
node_stg_customers -->|"feeds"| node_int_features
node_int_features -->|"curated into"| node_fct_analysis
node_stg_customers -->|"validated by"| node_schema_tests
node_int_features -->|"validated by"| node_schema_tests
node_fct_analysis -->|"validated by"| node_schema_tests
node_fct_analysis -->|"validated by"| node_custom_tests
node_custom_tests_yaml -->|"configures"| node_custom_tests
node_dbt_project -->|"includes"| node_seed_telco_churn
node_dbt_project -->|"includes"| node_stg_customers
node_dbt_project -->|"includes"| node_int_features
node_dbt_project -->|"includes"| node_fct_analysis
node_profiles -->|"connects"| node_dbt_project
node_airflow_dag -->|"runs"| node_dbt_project
node_github_ci -->|"runs"| node_dbt_project
node_compose -->|"builds"| node_dbt_dockerfile
node_compose -->|"builds"| node_airflow_dockerfile
node_dbt_dockerfile -->|"packages"| node_dbt_project
node_airflow_dockerfile -->|"packages"| node_airflow_dag
node_dbt_utils_pkg -->|"extends"| node_schema_tests

click node_data_raw_csv "https://github.com/javiersdr/churn-pipeline/blob/main/data/raw/WA_Fn-UseC_-Telco-Customer-Churn.csv"
click node_seed_telco_churn "https://github.com/javiersdr/churn-pipeline/blob/main/dbt_churn/seeds/telco_churn.csv"
click node_stg_customers "https://github.com/javiersdr/churn-pipeline/blob/main/dbt_churn/models/staging/stg_telco__customers.sql"
click node_int_features "https://github.com/javiersdr/churn-pipeline/blob/main/dbt_churn/models/intermediate/int_churn_features.sql"
click node_fct_analysis "https://github.com/javiersdr/churn-pipeline/blob/main/dbt_churn/models/marts/fct_churn_analysis.sql"
click node_schema_tests "https://github.com/javiersdr/churn-pipeline/blob/main/dbt_churn/models/schema.yml"
click node_custom_tests "https://github.com/javiersdr/churn-pipeline/blob/main/dbt_churn/tests/check_charges.sql"
click node_custom_tests_yaml "https://github.com/javiersdr/churn-pipeline/blob/main/dbt_churn/tests/check_charges.yml"
click node_dbt_project "https://github.com/javiersdr/churn-pipeline/blob/main/dbt_churn/dbt_project.yml"
click node_profiles "https://github.com/javiersdr/churn-pipeline/blob/main/dbt_churn/profiles.yml"
click node_airflow_dag "https://github.com/javiersdr/churn-pipeline/blob/main/airflow/dags/churn_pipeline.py"
click node_github_ci "https://github.com/javiersdr/churn-pipeline/blob/main/.github/workflows/dbt_ci.yml"
click node_compose "https://github.com/javiersdr/churn-pipeline/blob/main/docker-compose.yml"
click node_dbt_dockerfile "https://github.com/javiersdr/churn-pipeline/tree/main/Dockerfile"
click node_airflow_dockerfile "https://github.com/javiersdr/churn-pipeline/blob/main/Dockerfile.airflow"
click node_dbt_utils_pkg "https://github.com/javiersdr/churn-pipeline/tree/main/dbt_churn/dbt_packages/dbt_utils"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_data_raw_csv toneBlue
class node_seed_telco_churn,node_stg_customers,node_int_features,node_fct_analysis,node_schema_tests,node_custom_tests,node_custom_tests_yaml,node_dbt_project,node_profiles toneAmber
class node_airflow_dag,node_github_ci toneMint
class node_compose,node_dbt_dockerfile,node_airflow_dockerfile toneRose
class node_dbt_utils_pkg toneIndigo
```

## Components

### Raw data

External CSV input boundary.

- [Raw CSV](https://github.com/javiersdr/churn-pipeline/blob/main/data/raw/WA_Fn-UseC_-Telco-Customer-Churn.csv): source file · Original Telco churn dataset on disk. · `data/raw/WA_Fn-UseC_-Telco-Customer-Churn.csv`

### dbt pipeline

Transformation layers and quality checks.

- [Churn seed](https://github.com/javiersdr/churn-pipeline/blob/main/dbt_churn/seeds/telco_churn.csv): dbt seed · Seeded copy of the raw dataset loaded into DuckDB. · `dbt_churn/seeds/telco_churn.csv`
- [Staging](https://github.com/javiersdr/churn-pipeline/blob/main/dbt_churn/models/staging/stg_telco__customers.sql): dbt model · Source-alignment layer that renames, normalizes, and types customer fields. · `dbt_churn/models/staging/stg_telco__customers.sql`
- [Features](https://github.com/javiersdr/churn-pipeline/blob/main/dbt_churn/models/intermediate/int_churn_features.sql): dbt model · Intermediate churn feature engineering and derived metrics. · `dbt_churn/models/intermediate/int_churn_features.sql`
- [Churn fact](https://github.com/javiersdr/churn-pipeline/blob/main/dbt_churn/models/marts/fct_churn_analysis.sql): dbt mart · Final analysis-ready contract for reporting and downstream use. · `dbt_churn/models/marts/fct_churn_analysis.sql`
- [Schema tests](https://github.com/javiersdr/churn-pipeline/blob/main/dbt_churn/models/schema.yml): dbt tests · Model schema and YAML-based validation rules. · `dbt_churn/models/schema.yml`
- [Custom tests](https://github.com/javiersdr/churn-pipeline/blob/main/dbt_churn/tests/check_charges.sql): sql tests · Project-specific SQL checks for charge and phone logic. · `dbt_churn/tests/check_charges.sql`
- [Test config](https://github.com/javiersdr/churn-pipeline/blob/main/dbt_churn/tests/check_charges.yml): dbt test config · YAML configuration for custom SQL tests. · `dbt_churn/tests/check_charges.yml`
- [dbt project](https://github.com/javiersdr/churn-pipeline/blob/main/dbt_churn/dbt_project.yml): dbt config · dbt project root defining models, seeds, and runtime behavior. · `dbt_churn/dbt_project.yml`
- [Profiles](https://github.com/javiersdr/churn-pipeline/blob/main/dbt_churn/profiles.yml): dbt config · Local connection profile for DuckDB execution. · `dbt_churn/profiles.yml`

### Operations

Scheduling and CI control planes.

- [Airflow DAG](https://github.com/javiersdr/churn-pipeline/blob/main/airflow/dags/churn_pipeline.py): orchestrator · Daily scheduler that triggers the dbt pipeline. · `airflow/dags/churn_pipeline.py`
- [GitHub CI](https://github.com/javiersdr/churn-pipeline/blob/main/.github/workflows/dbt_ci.yml): workflow · Push-to-main validation workflow for dbt checks. · `.github/workflows/dbt_ci.yml`

### Runtime

Containerized local execution environment.

- [Compose](https://github.com/javiersdr/churn-pipeline/blob/main/docker-compose.yml): runtime manifest · Local multi-service setup tying dbt and Airflow together. · `docker-compose.yml`
- [dbt image](https://github.com/javiersdr/churn-pipeline/tree/main/Dockerfile): container build · Container definition for the dbt runtime. · `Dockerfile`
- [Airflow image](https://github.com/javiersdr/churn-pipeline/blob/main/Dockerfile.airflow): container build · Container definition for the Airflow scheduler runtime. · `Dockerfile.airflow`

### Vendored deps

Checked-in dbt package dependency layer.

- [dbt-utils](https://github.com/javiersdr/churn-pipeline/tree/main/dbt_churn/dbt_packages/dbt_utils): vendored package · Checked-in dependency providing dbt utility macros and generic tests. · `dbt_churn/dbt_packages/dbt_utils`

## Connections

- Raw CSV → Churn seed: ingested as · The raw file is loaded into the managed dbt seed table.
- Churn seed → Staging: modeled by · Staging reads from the seed and aligns the schema.
- Staging → Features: feeds · Clean customer records become input to feature engineering.
- Features → Churn fact: curated into · Intermediate logic rolls up into the final mart.
- Staging → Schema tests: validated by · Schema tests check the staging contract.
- Features → Schema tests: validated by · Schema tests also enforce intermediate model expectations.
- Churn fact → Schema tests: validated by · The mart is checked by model schema tests.
- Churn fact → Custom tests: validated by · Custom SQL tests verify business rules like charges and phone logic.
- Test config → Custom tests: configures · YAML defines the custom SQL test cases.
- dbt project → Churn seed: includes · Project configuration includes seed execution.
- dbt project → Staging: includes · Project configuration includes the staging model.
- dbt project → Features: includes · Project configuration includes the intermediate model.
- dbt project → Churn fact: includes · Project configuration includes the mart model.
- Profiles → dbt project: connects · Profiles supply the DuckDB connection target used by dbt.
- Airflow DAG → dbt project: runs · Airflow orchestrates dbt execution on a schedule.
- GitHub CI → dbt project: runs · CI invokes dbt validation on changes.
- Compose → dbt image: builds · Compose can build or run the dbt service image.
- Compose → Airflow image: builds · Compose can build or run the Airflow service image.
- dbt image → dbt project: packages · The dbt container carries the project runtime.
- Airflow image → Airflow DAG: packages · The Airflow container carries the scheduler and DAG.
- dbt-utils → Schema tests: extends · Vendored dbt-utils macros support generic tests and SQL helpers.
