How dbt-labs/dbt-cll-evaluator works
Project type and purpose
• This is a Python-based command-line application (CLI) whose sole purpose is to fetch, process, and display column-level lineage (CLL) information from dbt Cloud’s internal Discovery API. It pulls data for a single starting node and optionally applies “fix” operations against a local dbt project.
File structure
• Top-level:
• pyproject.toml → Python packaging (Poetry)
• README.md, LICENSE.md → documentation
• demo.cast, slides/ → presentation assets
• Source directory dbt_cll_evaluator/:
• __main__.py → CLI entry point (console_scripts)
• cll_types/ → domain models & data classes representing the lineage graph
• dbt_operations/ → modules that read/write a local dbt project (the “--fix” commands)
• rules.py → business rules on how to evaluate or transform lineage
README for additional insights
• Confirms it’s driven by poetry run dbt-cll-evaluator …
• Expects: dbt Cloud API token, environment ID, node_id for lineage extraction
• Two categories of operations:
a) “read” commands that call the Discovery API
b) “fix” commands that modify a target dbt project filesystem
Main components & interactions
a. CLI Layer
• Argument parser (built into __main__.py) that routes subcommands (e.g. “fetch‐cll”, “apply‐fix”)
b. API Client Layer
• A thin HTTP client module (likely in dbt_operations or a helper) that authenticates using the provided token, calls the Discovery API, and returns JSON
c. Domain Model Layer
• cll_types package: transforms raw JSON into strongly-typed lineage objects (nodes & edges)
d. Business Logic Layer
• rules.py: applies evaluation rules on the domain model (e.g. flagging lineage issues)
e. I/O & Persistence Layer
• For “read” commands: simply prints or streams to stdout
• For “--fix” commands: dbt_operations modules inspect & patch local dbt project files (parsing .sql/models, dbt_project.yml, etc.)
f. External Dependencies
• dbt Cloud Discovery API (external service)
• Local file system (the target dbt project path)
Architectural patterns & design principles
• Modular CLI design: clear separation of subcommands → easier extensibility
• Layered approach:
– Presentation (CLI)
– Service (API client)
– Domain (cll_types)
– Rules/Business logic
– Persistence (file system)
• Single Responsibility: each module has well-defined purpose (parsing, HTTP, domain modeling, rules, fs ops)