# pwwang/pipen architecture

> Architecture diagram of the GitHub repository pwwang/pipen, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-02). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/pwwang/pipen
- Repository: https://github.com/pwwang/pipen

## Overview

##### Main Components
- Pipeline Engine/Core – Represent the heart of the framework (see modules such as pipen/__main__.py, pipen/pipen.py, and pipen/job.py). This is where the orchestration happens.
- Process Management – The Proc and ProcGroup classes (pipen/proc.py, pipen/procgroup.py) manage individual processing units in a pipeline.
- Scheduler – The scheduler (pipen/scheduler.py) controls the execution and order of the processes.
- CLI Interface – A set of command-line modules found under pipen/cli (such as _main.py, help.py, and version.py) that allow users to interact with the framework.
- Plugin Manager – The pipen/pluginmgr.py module is dedicated to loading and managing different plugins that extend functionality (as indicated in the README’s plugin gallery).
- Channels, Templating, and Utility Modules – Other supporting modules (e.g., pipen/channel.py, pipen/template.py, pipen/utils.py, and pipen/progressbar.py) provide data flow, templating services, and logging/utility functions.
- Configuration & Environment – Include the configuration files like pyproject.toml, setup.py, along with continuous integration (CI) workflows (.github/workflows) and docs, which support deployment and documentation.

##### Relationships and Interactions
- The CLI acts as the entry point, allowing a user to run a pipeline (e.g., via run("MyPipeline", starts=P1)) which then instantiates a pipeline object from the core engine.
- The Pipeline Engine initializes the pipeline by loading processes and managing job scheduling.
- The Scheduler is responsible for determining the order of process execution and handling concurrency. The Engine sends jobs to the Scheduler, which then executes the processes.
- Upon initiation, the Engine constructs process instances (using the Proc classes) which are linked through dependencies (e.g., a Proc might require another Proc as its dependency).
- The Plugin Manager loads any available plugins (such as logging, CLI extension, caching, etc.) that modify or enhance behavior at runtime.
- Utility modules, including channels and templating, assist in data flow and dynamic script generation inside process definitions.

##### Architectural Patterns/Design Principles
- Modularity and Extensibility: Each key component (CLI, engine, scheduler, plugins) is loosely coupled. They communicate via well-defined interfaces; plugins are used to extend functionalities.
- Pipeline Composition Pattern: The design enables users to define pipelines with clear dependencies between processes.
- Dependency Injection & Configurability: Settings like caching, logging, and error handling are set at pipeline run time.
- A “nearly zero-configuration” approach is advertised by the framework, which should be reflected by how minimal components interact by default.

##### Relevant Technologies
- Python – The entire framework is built in Python.
- Third-party libraries for logging, templating, and testing (hinted at in the project’s tests folder and in the README examples).
- CI/CD workflows (.github/workflows) that manage build and documentation.
- Docker support via the provided Dockerfile in .codesandbox.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% CLI Interface Subgraph
    subgraph "CLI Interface"
        CLI_Main["_main.py"]:::cli
        CLI_Help["help.py"]:::cli
        CLI_Version["version.py"]:::cli
    end

    %% Pipeline Engine/Core Subgraph
    subgraph "Pipeline Engine/Core"
        Engine_Main["__main__.py"]:::engine
        Engine_Pipen["pipen.py"]:::engine
        Engine_Job["job.py"]:::engine
    end

    %% Process Management Subgraph
    subgraph "Process Management"
        Proc["proc.py"]:::process
        ProcGroup["procgroup.py"]:::process
    end

    %% Scheduler Node
    Scheduler["scheduler.py (Scheduler)"]:::scheduler

    %% Plugin Manager Node
    PluginManager["pluginmgr.py (Plugin Manager)"]:::plugin

    %% Utility Modules Subgraph
    subgraph "Utility Modules"
        Channel["channel.py"]:::util
        Template["template.py"]:::util
        Utils["utils.py"]:::util
        Progressbar["progressbar.py"]:::util
    end

    %% Configuration & Environment Subgraph
    subgraph "Configuration & Environment"
        Pyproject["pyproject.toml"]:::config
        Setup["setup.py"]:::config
        CI_Docs[".github/workflows & docs"]:::config
    end

    %% Connections
    CLI_Main -->|"initiates"| Engine_Main
    Engine_Main -->|"schedules"| Scheduler
    Engine_Main -->|"creates"| Proc
    Scheduler -->|"executes"| Proc
    Engine_Main <-->|"manages"| PluginManager
    Engine_Main -->|"utilizes"| Channel
    Proc -->|"communicates"| Channel
    Pyproject -->|"configures"| Engine_Main
    Setup -->|"configures"| Engine_Main
    CI_Docs -->|"supports"| Engine_Main

    %% Click Events for Pipeline Engine/Core
    click Engine_Main "https://github.com/pwwang/pipen/blob/master/pipen/__main__.py"
    click Engine_Pipen "https://github.com/pwwang/pipen/blob/master/pipen/pipen.py"
    click Engine_Job "https://github.com/pwwang/pipen/blob/master/pipen/job.py"

    %% Click Events for Process Management
    click Proc "https://github.com/pwwang/pipen/blob/master/pipen/proc.py"
    click ProcGroup "https://github.com/pwwang/pipen/blob/master/pipen/procgroup.py"

    %% Click Event for Scheduler
    click Scheduler "https://github.com/pwwang/pipen/blob/master/pipen/scheduler.py"

    %% Click Events for CLI Interface
    click CLI_Main "https://github.com/pwwang/pipen/blob/master/pipen/cli/_main.py"
    click CLI_Help "https://github.com/pwwang/pipen/blob/master/pipen/cli/help.py"
    click CLI_Version "https://github.com/pwwang/pipen/blob/master/pipen/cli/version.py"

    %% Click Event for Plugin Manager
    click PluginManager "https://github.com/pwwang/pipen/blob/master/pipen/pluginmgr.py"

    %% Click Events for Utility Modules
    click Channel "https://github.com/pwwang/pipen/blob/master/pipen/channel.py"
    click Template "https://github.com/pwwang/pipen/blob/master/pipen/template.py"
    click Utils "https://github.com/pwwang/pipen/blob/master/pipen/utils.py"
    click Progressbar "https://github.com/pwwang/pipen/blob/master/pipen/progressbar.py"

    %% Click Events for Configuration & Environment
    click Pyproject "https://github.com/pwwang/pipen/blob/master/pyproject.toml"
    click Setup "https://github.com/pwwang/pipen/blob/master/setup.py"
    click CI_Docs "https://github.com/pwwang/pipen/tree/master/.github/workflows"

    %% Styles
    classDef cli fill:#F7C6C7,stroke:#A62349,stroke-width:2px;
    classDef engine fill:#C2E7DA,stroke:#2C7A7B,stroke-width:2px;
    classDef process fill:#F9E79F,stroke:#B7950B,stroke-width:2px;
    classDef scheduler fill:#D5F5E3,stroke:#28B463,stroke-width:2px;
    classDef plugin fill:#D6EAF8,stroke:#2471A3,stroke-width:2px;
    classDef util fill:#FAD7A0,stroke:#D68910,stroke-width:2px;
    classDef config fill:#E8DAEF,stroke:#8E44AD,stroke-width:2px;
```
