# bl3f/yato architecture

> Architecture diagram of the GitHub repository bl3f/yato, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-21). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/bl3f/yato
- Repository: https://github.com/bl3f/yato

## Overview

##### File Structure Analysis
- yato/ (core package)
- cli.py        – command-line entrypoint
- yato.py       – high-level Yato class with run/backup/restore
- parser.py     – builds dependency graph via sqlglot
- storage.py    – reads/writes SQL files, env‐vars, and DuckDB I/O
- python_context.py – handles Python‐based transformations
- cli.py (root)  – thin wrapper to call yato.cli
- examples/      – sample folders with .sql/.mmd/.py to illustrate use
- tests/         – unit tests for parser, storage, CLI, core logic
- pyproject.toml – packaging, dependencies
- GitHub Actions – CI workflow (lint & tests)

##### Core Components and Interactions
- CLI Layer (cli.py)
- Parses command line args (subcommand “run”, “backup”, etc.)
- Instantiates Yato class
- Shows logs / error handling
- Yato Core (yato/yato.py)
- Reads SQL/py files via Storage
- Hands each file to Parser to extract source/target table & build DAG
- Executes nodes in topological order:
– For SQL: run statements in DuckDB via Python API
– For Python: load pandas DataFrame, run user’s Transformation.run(), write back
- Manages backup/restore around dlt integration
- Parser (yato/parser.py)
- Leverages SQLGlot to parse queries, find SELECT dependencies
- Builds in-memory DAG (nodes = transformation units)
- Storage (yato/storage.py)
- Reads file system tree under sql_folder
- Applies environment variable substitution
- Persists/loads DuckDB database files
- Python Context (yato/python_context.py)
- Dynamically imports user-defined Transformations
- Provides context for DataFrame operations

##### Architectural Patterns & Design Principles
- Modular Layering: CLI → Core Orchestrator → Parser/Storage/Execution modules
- Plugin-style Python Transformations: users can drop .py with a `Transformation` subclass
- Configuration-driven: folder structure drives DAG, minimal conventions over configuration
- Single-responsibility: each module (parser, storage, execution) has a focused role
- Dependency Injection (thin): Storage and Parser are passed into Yato core

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Core Layers
    CLI_Launch["CLI Entrypoint\ncli.py"]:::cli
    Yato_CLI["CLI Layer\nyato/cli.py"]:::cli
    Yato_Core["Yato Orchestrator\nyato/yato.py"]:::core
    Parser["Parser Engine\nyato/parser.py"]:::core
    Storage["Storage Manager\nyato/storage.py"]:::io
    SQLExec["SQL Executor\nDuckDB Python API"]:::io
    DuckDB["DuckDB Database"]:::io
    PythonCtx["Python Transformer\nyato/python_context.py"]:::ext
    MermaidGen["Mermaid Generator\nyato/mermaid.py"]:::gen

    %% Out-of-band Code
    subgraph "Out-of-band Code"
        Examples["Examples\nexamples/"]:::outof
        Tests["Tests\ntests/"]:::outof
        Pyproject["pyproject.toml"]:::outof
        CI["GitHub Actions\n.github/workflows/main.yml"]:::outof
    end

    %% Flows
    CLI_Launch -->|"invokes"| Yato_CLI
    Yato_CLI -->|"instantiates"| Yato_Core
    Yato_Core -->|"reads/writes files"| Storage
    Yato_Core -->|"builds DAG"| Parser
    Yato_Core -->|"executes SQL"| SQLExec
    SQLExec -->|"uses"| DuckDB
    Yato_Core -->|"executes Python"| PythonCtx
    Yato_Core -->|"generates diagram"| MermaidGen
    MermaidGen -->|"outputs .mmd"| Examples

    %% Click Events
    click CLI_Launch "https://github.com/bl3f/yato/blob/main/cli.py"
    click Yato_CLI "https://github.com/bl3f/yato/blob/main/yato/cli.py"
    click Yato_Core "https://github.com/bl3f/yato/blob/main/yato/yato.py"
    click Parser "https://github.com/bl3f/yato/blob/main/yato/parser.py"
    click Storage "https://github.com/bl3f/yato/blob/main/yato/storage.py"
    click SQLExec "https://github.com/bl3f/yato/blob/main/yato/storage.py"
    click DuckDB "https://github.com/bl3f/yato/blob/main/yato/storage.py"
    click PythonCtx "https://github.com/bl3f/yato/blob/main/yato/python_context.py"
    click MermaidGen "https://github.com/bl3f/yato/blob/main/yato/mermaid.py"
    click Examples "https://github.com/bl3f/yato/tree/main/examples/"
    click Tests "https://github.com/bl3f/yato/tree/main/tests/"
    click Pyproject "https://github.com/bl3f/yato/blob/main/pyproject.toml"
    click CI "https://github.com/bl3f/yato/blob/main/.github/workflows/main.yml"

    %% Styles
    classDef io fill:#e0f7fa,stroke:#00838f,stroke-width:1px
    classDef core fill:#bbdefb,stroke:#0d47a1,stroke-width:1px
    classDef ext fill:#fff9c4,stroke:#f9a825,stroke-width:1px
    classDef gen fill:#d1c4e9,stroke:#4a148c,stroke-width:1px
    classDef cli fill:#ffcdd2,stroke:#b71c1c,stroke-width:1px
    classDef outof fill:#eeeeee,stroke-dasharray: 5 5
```
