How rlado/canonada works
Project type and purpose
• Canonada is an open-source Python CLI framework and library for building, visualizing, and running memory-efficient streaming data pipelines.
File structure
• Top-level files (pyproject.toml, requirements.txt, Makefile, mypy.ini, LICENSE) set up build, linting, packaging, and CI (.github/workflows).
• src/canonada/cli.py is the entry point for all commands (new, catalog, registry, run, view, version).
• Core subpackages:
• src/canonada/_config.py – loads and validates project‐level TOML settings (catalog, credentials, parameters).
• src/canonada/catalog/ – defines CatalogCore and datahandler plugins (_datahandlers.py).
• src/canonada/pipeline/ – defines PipelineCore, Node, and streaming execution engine.
• src/canonada/system/ – defines SystemCore, orchestration of multiple pipelines.
• src/canonada/_logger.py, _utils/ and _version.py for logging, helpers, and versioning.
• src/canonada/templates/ holds file and directory templates for scaffolding new projects (config files, pipelines, systems, datahandler boilerplate).
• Tests mirror a sample project under tests/basic_test_project/ and unit tests for core modules.
README for additional insights
• Highlights the modular, memory-efficient streaming pipeline model.
• Shows how users define pipelines as graphs of Node functions tied to the catalog and parameters.
• Emphasizes commands: canonada new, canonada run/view pipelines|systems, canonada catalog, canonada registry.
Detail and modularity
• Subpackage as its own component.
• TOML configuration files as inputs to the Config Manager.
• Don’t shy away from showing both the “scaffold a new project” path and the “execute a pipeline/system” path side by side—this reflects Canonada’s dual nature as both a generator and a runtime framework.