# fszta/dbt-column-lineage architecture

> Architecture diagram of the GitHub repository fszta/dbt-column-lineage, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-02). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/fszta/dbt-column-lineage
- Repository: https://github.com/fszta/dbt-column-lineage

## Overview

##### Main Components
- Command-Line Interface (CLI):
– Located in the dbt_column_lineage/cli/main.py file.
– Responsible for parsing user input, command-line options, and triggering the overall process.
- Artifacts Processing Module:
– Found in the dbt_column_lineage/artifacts directory (files such as catalog.py, manifest.py, sql_parser.py).
– Responsible for reading, parsing, and possibly validating the dbt manifest and catalog JSON files.
- Lineage Computation Service:
– Implemented in dbt_column_lineage/lineage/service.py along with models/schema.py in the models subdirectory.
– Combines information from the artifacts with additional parsing (using SQL parsing in artifacts) to compute column lineage relationships.
- /Output Modules:
– They are separated by format:
▸ HTML: dbt_column_lineage/lineage/display/html/ (with display.py, static assets, and templates)
▸ DOT: dbt_column_lineage/lineage/display/dot.py
▸ Text: dbt_column_lineage/lineage/display/text.py
– These modules take the computed lineage data and format it for the user.
- Testing Components:
– Located in both the tests/integration and tests/unit directories.
– Provide integration and unit tests ensuring the correctness of the components (CLI, artifacts parsing, lineage computation, and display formats).

##### Relationships and Data Flow
- When the user invokes a command via the CLI, the input arguments (especially the file paths for manifest and catalog, which are prerequisites according to the README) are received by main.py.
- The CLI then calls into the Artifacts Processing Module to load and parse the dbt files.
- Parsed data is handed off to the Lineage Computation Service which uses both the metadata in the artifacts and SQL parsing (via SQL parser in artifacts/sql_parser.py) to determine upstream and downstream relationships.
- The computed data is then passed to one of the Display Modules depending on the command-line option (--format) chosen by the user:
▸ HTML for an interactive web interface (which even spins up a server on a specified port)
▸ DOT for GraphViz output
▸ Text for simple console output.
- Each of these components works in a pipeline fashion where the CLI is the entry point and display modules provide the final output.

##### Architectural Patterns and Design Principles
- Separation of Concerns:
– There is clear separation between input (CLI), processing (artifacts, lineage service, SQL parsing), and output (display modules).
- Modular Design:
– The project is divided into dedicated modules for parsing, computing, and displaying lineage data, which aids in testing and maintenance.
- Extensibility:
– The design allows adding new display formats or artifact processors if needed.

##### Relevant Technologies and Frameworks
- Python as the main development language (shown by the usage of pyproject.toml and typical Python project structure).
- dbt artifacts which are JSON files generated by dbt tools.
- GraphViz (via DOT output) and a simple HTTP server for the HTML view.
- Testing frameworks as indicated by the tests setup.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% CLI Layer
    CLI["CLI Interface"]:::cli

    %% Artifacts Processing Module
    subgraph "Artifacts Processing Module"
        A1["Catalog Parser"]:::artifacts
        A2["Manifest Parser"]:::artifacts
        A3["SQL Parser"]:::artifacts
    end

    %% Aggregated Artifact Data
    A_AGG["Artifacts Data"]:::artifacts

    %% Lineage Computation Service
    subgraph "Lineage Computation Service"
        LC1["Lineage Service"]:::service
        LC2["Schema Models"]:::service
    end

    %% Display Modules
    subgraph "Display Modules"
        D1["HTML Display"]:::display
        D2["DOT Display"]:::display
        D3["Text Display"]:::display
    end

    %% Testing Components
    subgraph "Testing Components"
        T1["Integration Tests"]:::testing
        T2["Unit Tests"]:::testing
    end

    %% Main Data Flow
    CLI -->|"parses input"| A1
    CLI -->|"parses input"| A2
    CLI -->|"parses input"| A3
    A1 --> A_AGG
    A2 --> A_AGG
    A3 --> A_AGG
    A_AGG -->|"provides artifacts"| LC1
    LC1 --> LC2
    LC2 -->|"formats lineage"| D1
    LC2 -->|"formats lineage"| D2
    LC2 -->|"formats lineage"| D3

    %% Testing Connections (dotted lines)
    CLI --- T1
    CLI --- T2
    A_AGG --- T1
    A_AGG --- T2
    LC2 --- T1
    LC2 --- T2
    D1 --- T1
    D1 --- T2

    %% Click Events
    click CLI "https://github.com/fszta/dbt-column-lineage/blob/main/dbt_column_lineage/cli/main.py"
    click A1 "https://github.com/fszta/dbt-column-lineage/blob/main/dbt_column_lineage/artifacts/catalog.py"
    click A2 "https://github.com/fszta/dbt-column-lineage/blob/main/dbt_column_lineage/artifacts/manifest.py"
    click A3 "https://github.com/fszta/dbt-column-lineage/blob/main/dbt_column_lineage/artifacts/sql_parser.py"
    click LC1 "https://github.com/fszta/dbt-column-lineage/blob/main/dbt_column_lineage/lineage/service.py"
    click LC2 "https://github.com/fszta/dbt-column-lineage/blob/main/dbt_column_lineage/models/schema.py"
    click D1 "https://github.com/fszta/dbt-column-lineage/blob/main/dbt_column_lineage/lineage/display/html/display.py"
    click D2 "https://github.com/fszta/dbt-column-lineage/blob/main/dbt_column_lineage/lineage/display/dot.py"
    click D3 "https://github.com/fszta/dbt-column-lineage/blob/main/dbt_column_lineage/lineage/display/text.py"
    click T1 "https://github.com/fszta/dbt-column-lineage/tree/main/tests/integration"
    click T2 "https://github.com/fszta/dbt-column-lineage/tree/main/tests/unit"

    %% Styles
    classDef cli fill:#ffcccc,stroke:#333,stroke-width:2px;
    classDef artifacts fill:#ccffcc,stroke:#333,stroke-width:2px;
    classDef service fill:#ccccff,stroke:#333,stroke-width:2px;
    classDef display fill:#ffffcc,stroke:#333,stroke-width:2px;
    classDef testing fill:#ffe6cc,stroke:#333,stroke-dasharray: 5 5,stroke-width:2px;
```
