# rlado/canonada architecture

> Architecture diagram of the GitHub repository rlado/canonada, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-21). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/rlado/canonada
- Repository: https://github.com/rlado/canonada

## Overview

##### Project type and purpose
- Canonada is an open-source Python CLI framework and library for building, visualizing, and running memory-efficient streaming data pipelines.

##### File structure
- Top-level files (pyproject.toml, requirements.txt, Makefile, mypy.ini, LICENSE) set up build, linting, packaging, and CI (`.github/workflows`).
- `src/canonada/cli.py` is the entry point for all commands (`new`, `catalog`, `registry`, `run`, `view`, `version`).
- Core subpackages:
- `src/canonada/_config.py` – loads and validates project‐level TOML settings (catalog, credentials, parameters).
- `src/canonada/catalog/` – defines `CatalogCore` and datahandler plugins (`_datahandlers.py`).
- `src/canonada/pipeline/` – defines `PipelineCore`, `Node`, and streaming execution engine.
- `src/canonada/system/` – defines `SystemCore`, orchestration of multiple pipelines.
- `src/canonada/_logger.py`, `_utils/` and `_version.py` for logging, helpers, and versioning.
- `src/canonada/templates/` holds file and directory templates for scaffolding new projects (config files, pipelines, systems, datahandler boilerplate).
- Tests mirror a sample project under `tests/basic_test_project/` and unit tests for core modules.

##### README for additional insights
- Highlights the modular, memory-efficient streaming pipeline model.
- Shows how users define pipelines as graphs of `Node` functions tied to the catalog and parameters.
- Emphasizes commands: `canonada new`, `canonada run/view pipelines|systems`, `canonada catalog`, `canonada registry`.

##### Detail and modularity
- Subpackage as its own component.
- TOML configuration files as inputs to the Config Manager.
- Don’t shy away from showing both the “scaffold a new project” path and the “execute a pipeline/system” path side by side—this reflects Canonada’s dual nature as both a generator and a runtime framework.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% CLI Layer
    subgraph "CLI Layer"
        CLI["CLI Layer\n(cli.py)"]:::cli
        Version["Version Command\n(_version.py)"]:::cli
    end

    %% Core Services Layer
    subgraph "Core Services"
        Config["Config Manager\n(_config.py)"]:::core
        Logger["Logger Service\n(_logger.py)"]:::core
        Utils["Utility Helpers\n(progressbar.py)"]:::core
    end

    %% Catalog Service Layer
    subgraph "Catalog Service"
        CatalogCore["Catalog Core\n(_core.py)"]:::catalog
        DataHandlers["Datahandlers Registry\n(_datahandlers.py)"]:::catalog
    end

    %% Pipeline & System Engines
    subgraph "Pipeline & System Engines"
        subgraph "Pipeline Engine"
            PipelineCore["Pipeline Engine Core\n(_core.py)"]:::engine
        end
        subgraph "System Orchestrator"
            SystemCore["System Orchestrator Core\n(_core.py)"]:::engine
        end
    end

    %% Template Engine Layer
    subgraph "Template Engine"
        TOMLTemplate["Project TOML Template\n(canonada.toml)"]:::template
        ConfigTemplates["Config Templates\n(config/)"]:::template
        DatahandlerTemplates["Datahandler Boilerplate\n(datahandlers/)"]:::template
        PipelineTemplates["Pipeline Templates\n(pipelines/)"]:::template
        SystemTemplates["System Templates\n(systems/)"]:::template
    end

    %% External Resources
    subgraph "External Resources"
        FS["File System\n(TOML, CSV, JSON, Parquet)"]:::external
        ExtServices["External Services/DBs"]:::external
    end

    %% User Extension Points
    subgraph "User Extension Points"
        UserPipelines["User Pipelines\n(functions)"]:::user
        UserDatahandlers["User Datahandlers"]:::user
        UserSystems["User Systems"]:::user
    end

    %% Connections
    CLI -->|"load settings"| Config
    CLI -->|"run/view pipelines"| PipelineCore
    CLI -->|"run/view systems"| SystemCore
    CLI -->|"manage catalog"| CatalogCore
    CLI -->|"manage registry"| DataHandlers
    CLI -->|"new project"| TOMLTemplate
    CLI --> Version

    Config --> CatalogCore
    CatalogCore --> DataHandlers
    PipelineCore <-->|"fetch/write data"| CatalogCore
    PipelineCore -->|"log progress"| Logger
    PipelineCore -->|"use helpers"| Utils
    SystemCore -->|"orchestrate"| PipelineCore

    PipelineCore <--> UserPipelines
    CatalogCore <--> UserDatahandlers
    SystemCore <--> UserSystems

    DataHandlers --> FS
    CatalogCore --> FS
    PipelineCore --> ExtServices

    TOMLTemplate --> ConfigTemplates
    TOMLTemplate --> DatahandlerTemplates
    TOMLTemplate --> PipelineTemplates
    TOMLTemplate --> SystemTemplates
    TOMLTemplate --> FS

    %% Click Events
    click CLI "https://github.com/rlado/canonada/blob/master/src/canonada/cli.py"
    click Config "https://github.com/rlado/canonada/blob/master/src/canonada/_config.py"
    click Logger "https://github.com/rlado/canonada/blob/master/src/canonada/_logger.py"
    click Utils "https://github.com/rlado/canonada/blob/master/src/canonada/_utils/progressbar.py"
    click Version "https://github.com/rlado/canonada/blob/master/src/canonada/_version.py"
    click CatalogCore "https://github.com/rlado/canonada/blob/master/src/canonada/catalog/_core.py"
    click DataHandlers "https://github.com/rlado/canonada/blob/master/src/canonada/catalog/_datahandlers.py"
    click PipelineCore "https://github.com/rlado/canonada/blob/master/src/canonada/pipeline/_core.py"
    click SystemCore "https://github.com/rlado/canonada/blob/master/src/canonada/system/_core.py"
    click TOMLTemplate "https://github.com/rlado/canonada/blob/master/src/canonada/templates/canonada.toml"
    click ConfigTemplates "https://github.com/rlado/canonada/tree/master/src/canonada/templates/config/"
    click DatahandlerTemplates "https://github.com/rlado/canonada/tree/master/src/canonada/templates/datahandlers/"
    click PipelineTemplates "https://github.com/rlado/canonada/tree/master/src/canonada/templates/pipelines/"
    click SystemTemplates "https://github.com/rlado/canonada/tree/master/src/canonada/templates/systems/"

    %% Styles
    classDef cli fill:#ADD8E6,stroke:#000,stroke-width:1px
    classDef core fill:#ADD8E6,stroke:#000,stroke-width:1px
    classDef catalog fill:#ADD8E6,stroke:#000,stroke-width:1px
    classDef engine fill:#90EE90,stroke:#000,stroke-width:1px
    classDef template fill:#FFA500,stroke:#000,stroke-width:1px
    classDef external fill:#D3D3D3,stroke:#000,stroke-width:1px
    classDef user fill:none,stroke:#000,stroke-dasharray:5 5,stroke-width:1px
```
