How bl3f/yato works
File Structure Analysis
• yato/ (core package)
• cli.py – command-line entrypoint
• yato.py – high-level Yato class with run/backup/restore
• parser.py – builds dependency graph via sqlglot
• storage.py – reads/writes SQL files, env‐vars, and DuckDB I/O
• python_context.py – handles Python‐based transformations
• cli.py (root) – thin wrapper to call yato.cli
• examples/ – sample folders with .sql/.mmd/.py to illustrate use
• tests/ – unit tests for parser, storage, CLI, core logic
• pyproject.toml – packaging, dependencies
• GitHub Actions – CI workflow (lint & tests)
Core Components and Interactions
• CLI Layer (cli.py)
• Parses command line args (subcommand “run”, “backup”, etc.)
• Instantiates Yato class
• Shows logs / error handling
• Yato Core (yato/yato.py)
• Reads SQL/py files via Storage
• Hands each file to Parser to extract source/target table & build DAG
• Executes nodes in topological order:
– For SQL: run statements in DuckDB via Python API
– For Python: load pandas DataFrame, run user’s Transformation.run(), write back
• Manages backup/restore around dlt integration
• Parser (yato/parser.py)
• Leverages SQLGlot to parse queries, find SELECT dependencies
• Builds in-memory DAG (nodes = transformation units)
• Storage (yato/storage.py)
• Reads file system tree under sql_folder
• Applies environment variable substitution
• Persists/loads DuckDB database files
• Python Context (yato/python_context.py)
• Dynamically imports user-defined Transformations
• Provides context for DataFrame operations
Architectural Patterns & Design Principles
• Modular Layering: CLI → Core Orchestrator → Parser/Storage/Execution modules
• Plugin-style Python Transformations: users can drop .py with a Transformation subclass
• Configuration-driven: folder structure drives DAG, minimal conventions over configuration
• Single-responsibility: each module (parser, storage, execution) has a focused role
• Dependency Injection (thin): Storage and Parser are passed into Yato core