How reata/sqllineage works
Main Components
a) Core Processing Components:
• SQL Parser Layer (using SQLFluff and SQLParse)
• Lineage Analyzer
• Graph Storage (using NetworkX)
• Metadata Provider
• Runner/Executor
b) Interface Components:
• CLI Interface
• Web Visualization Interface (React-based)
• API Layer
Component Relationships
[User Interfaces]
• CLI (sqllineage.cli)
• Web UI (sqllineagejs)
[Core Processing]
• Runner (sqllineage.runner)
• Analyzer (core.analyzer)
• Parser Layer
• SQLFluff Parser
• SQLParse Parser
• Metadata Layer
• SQLAlchemy Provider
• Dummy Provider
• Graph Storage
• NetworkX
[External Dependencies]
• SQLAlchemy (for metadata)
• SQLFluff (for parsing)
• NetworkX (for graph representation)
Key Workflows to Highlight
a) SQL Processing Flow:
Input SQL → Parser → Analyzer → Graph Storage → Output (CLI/Visual)
b) Metadata Flow:
Database → SQLAlchemy Provider → Metadata Layer → Analyzer
Important Details
• Pluggable nature of the parser system (SQLFluff/SQLParse)
• Metadata provider's role in enhancing lineage accuracy
• Demonstrate the separation between core processing and visualization
• Column-level and table-level lineage paths