How fszta/dbt-column-lineage works
Main Components
• Command-Line Interface (CLI):
– Located in the dbt_column_lineage/cli/main.py file.
– Responsible for parsing user input, command-line options, and triggering the overall process.
• Artifacts Processing Module:
– Found in the dbt_column_lineage/artifacts directory (files such as catalog.py, manifest.py, sql_parser.py).
– Responsible for reading, parsing, and possibly validating the dbt manifest and catalog JSON files.
• Lineage Computation Service:
– Implemented in dbt_column_lineage/lineage/service.py along with models/schema.py in the models subdirectory.
– Combines information from the artifacts with additional parsing (using SQL parsing in artifacts) to compute column lineage relationships.
• /Output Modules:
– They are separated by format:
▸ HTML: dbt_column_lineage/lineage/display/html/ (with display.py, static assets, and templates)
▸ DOT: dbt_column_lineage/lineage/display/dot.py
▸ Text: dbt_column_lineage/lineage/display/text.py
– These modules take the computed lineage data and format it for the user.
• Testing Components:
– Located in both the tests/integration and tests/unit directories.
– Provide integration and unit tests ensuring the correctness of the components (CLI, artifacts parsing, lineage computation, and display formats).
Relationships and Data Flow
• When the user invokes a command via the CLI, the input arguments (especially the file paths for manifest and catalog, which are prerequisites according to the README) are received by main.py.
• The CLI then calls into the Artifacts Processing Module to load and parse the dbt files.
• Parsed data is handed off to the Lineage Computation Service which uses both the metadata in the artifacts and SQL parsing (via SQL parser in artifacts/sql_parser.py) to determine upstream and downstream relationships.
• The computed data is then passed to one of the Display Modules depending on the command-line option (--format) chosen by the user:
▸ HTML for an interactive web interface (which even spins up a server on a specified port)
▸ DOT for GraphViz output
▸ Text for simple console output.
• Each of these components works in a pipeline fashion where the CLI is the entry point and display modules provide the final output.
Architectural Patterns and Design Principles
• Separation of Concerns:
– There is clear separation between input (CLI), processing (artifacts, lineage service, SQL parsing), and output (display modules).
• Modular Design:
– The project is divided into dedicated modules for parsing, computing, and displaying lineage data, which aids in testing and maintenance.
• Extensibility:
– The design allows adding new display formats or artifact processors if needed.
Relevant Technologies and Frameworks
• Python as the main development language (shown by the usage of pyproject.toml and typical Python project structure).
• dbt artifacts which are JSON files generated by dbt tools.
• GraphViz (via DOT output) and a simple HTTP server for the HTML view.
• Testing frameworks as indicated by the tests setup.