# coder-na-r1/pipeline architecture

> Architecture diagram of the GitHub repository coder-na-r1/pipeline, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-07). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/coder-na-r1/pipeline
- Repository: https://github.com/coder-na-r1/pipeline

## Overview

##### Project Type and Purpose
- This is a framework for building pipelines that process multidimensional data (e.g., tensors, digital images).
- It supports both code-based and configuration file–based pipeline assembly.
- Key design goals include maximum performance, module decoupling (via dependency inversion and “concepts”), and cross-platform support (Linux and Windows).
- External dependencies include OpenCV, xtensor, CapnProto, cppzmq, and Boost.

##### File Structure for Architectural Clues
- The top-level directories are organized by purpose:
- Directories like src, include, and test contain the core framework code.
- Subdirectories in src (e.g., builder, connector, dLManager, monitoringManager, networkBroker, networkManager, parser, pipeline, protoManager, scalableManager, stageManager, utils) indicate that the system is modularized by functional concerns.
- The builder subdirectory is further split into “parallelBuilder” and “scalableBuilder,” which demonstrates the use of the Builder pattern for constructing pipeline configurations of different types.
- The connector modules (found under both src/connector and src/detail/connector) suggest the framework supports different types of data or control connectors (e.g., SPSCConnector, MPMCConnector) between pipeline stages.
- Directories like benchmark, examples, necessary, and dataStream indicate that sample pipelines, performance benchmarks, stream handling (with associated CapnProto schema files), and even Python interfacing are provided.
- The extensive include directory (with libraries under “cds” and “json”) implies heavy use of concurrent data structures and external libraries.

##### README for Additional Insights
- The README clearly states that the framework is meant for building pipelines to process multidimensional data.
- It lays out design goals (performance, low coupling, cross-platform) that reinforce the modular, high-performance and configuration-driven nature of the system.
- The build instructions (using CMake) further confirm the structure is meant to be portable and modular.

##### Main System Components
a. Pipeline Assembly / Builder Module:
- Separate “ParallelPipelineBuilder”, “ScalablePipelineBuilder,” and their underlying interfaces (e.g., IStageBuilder, IScalableStageBuilder, IScalableManagerBuilder).
- How these builders take configuration files (e.g., PipeConfig.json, ScalablePipeConfig.json) and code inputs to generate a pipeline.
b. Pipeline Stages and Connectors:
- Stage components, including the “Stage” abstraction (src/pipeline/Stage.hpp) along with the various connectors (src/connector and src/detail/connector) such as SPSCConnector, MPMCConnector, and SyncMapConnector.
c. Dynamic Linking Manager (dLManager):
- DLManager and its related interfaces (IDLManager) which handle dynamic linking loading logic (also tied to the “createDL” module).
d. Monitoring and Measurement System:
- Components within monitoringManager and src/detail/monitoring (e.g., IMonitoringManager, MonitoringMeasurements) that are used to monitor pipeline performance.
e. Network Components:
- Network layer with networkBroker and networkManager modules (e.g., ZmqReceiverBroker, ZmqSendlerBroker) which seem responsible for communicating across processes or machines.
f. Parser / Configuration Loader:
g. Data Streams and Schemas:
- DataStream module along with its CapnProto schema (ndarray.capnp files) to indicate how multidimensional data is structured and parsed.
h. External Dependencies and Libraries:
- Dependencies such as OpenCV, xtensor, CapnProto, cppzmq, Boost and concurrent data structures (the extensive “cds” directory) as supporting or underlying technologies.

##### Relationships and Interactions
- The parser reads configuration files (e.g., PipeConfig.json) and provides instructions to the builder module.
- The builder module assembles a series of pipeline stages; these stages connect via connectors that control the data flow on a possibly concurrent basis (using concurrent data structures).
- The dynamic linking (DLManager) component may load external modules (plugins) at runtime, which then get integrated into the pipeline.
- Monitoring and network modules interact with the core pipeline to capture performance metrics and support distributed processing, respectively.
- Benchmark and examples components, while not part of the core runtime, illustrate usage and allow performance evaluation.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Configuration Section
    subgraph "Configuration"
        CF["Configuration Files (PipeConfig.json, etc.)"]:::config
        P["Parser (Parser, DLParser, etc.)"]:::parser
    end

    %% Pipeline Builder Section
    subgraph "Pipeline Builder"
        PPB["Parallel Pipeline Builder"]:::builder
        SPB["Scalable Pipeline Builder"]:::builder
    end

    %% Pipeline Execution Section
    subgraph "Pipeline Execution"
        PS["Pipeline Stage"]:::execution
        CON["Connectors (SPSC, MPMC, SyncMap)"]:::execution
    end

    %% Dynamic Linking Section
    subgraph "Dynamic Linking"
        DL["Dynamic Linking Manager (DLManager)"]:::dynamic
    end

    %% Monitoring and Network Section
    subgraph "Monitoring & Network"
        MM["Monitoring Manager"]:::monitoring
        NTB["Network Broker"]:::network
        NTM["Network Manager"]:::network
    end

    %% Data Streams and Schemas Section
    subgraph "Data Streams & Schemas"
        DS["DataStream Module"]:::datastream
        CPS["CapnProto Schema (ndarray.capnp)"]:::datastream
    end

    %% External Dependencies Section
    subgraph "External Dependencies"
        EX1["OpenCV, xtensor, CapnProto, cppzmq, Boost"]:::external
        CDS["Concurrent Data Structures (cds)"]:::external
        JSON["JSON Library"]:::external
    end

    %% Usage and Testing Section
    subgraph "Usage & Testing"
        UT["Benchmark/Examples/Test"]:::test
    end

    %% Relationships and Flow
    CF --> P
    P --> PPB
    P --> SPB
    PPB --> PS
    SPB --> PS
    PS --> CON
    DL -- "integrates" --> PPB
    DL -- "supplements" --> CON
    MM -- "monitors" --> PS
    MM -- "monitors" --> CON
    NTB -- "supports" --> PS
    NTM -- "supports" --> CON
    DS -- "supplies data" --> PS
    CPS -- "defines" --> DS
    EX1 --- PPB
    EX1 --- SPB
    CDS --- CON
    JSON --- P
    UT --- PS

    %% Styles
    classDef config fill:#F4D03F,stroke:#333,stroke-width:2px;
    classDef parser fill:#85C1E9,stroke:#333,stroke-width:2px;
    classDef builder fill:#F5B7B1,stroke:#333,stroke-width:2px;
    classDef execution fill:#ABEBC6,stroke:#333,stroke-width:2px;
    classDef dynamic fill:#D2B4DE,stroke:#333,stroke-width:2px;
    classDef monitoring fill:#F9E79F,stroke:#333,stroke-width:2px;
    classDef network fill:#A9CCE3,stroke:#333,stroke-width:2px;
    classDef datastream fill:#FAD7A0,stroke:#333,stroke-width:2px;
    classDef external fill:#D5F5E3,stroke:#333,stroke-width:2px;
    classDef test fill:#E8DAEF,stroke:#333,stroke-width:2px;

    %% Click Events
    click PPB "https://github.com/coder-na-r1/pipeline/blob/main/src/builder/parallelBuilder/ParallelPipelineBuilder.hpp"
    click SPB "https://github.com/coder-na-r1/pipeline/blob/main/src/builder/scalableBuilder/ScalableParallelPipelineBuilder.hpp"
    click PS "https://github.com/coder-na-r1/pipeline/blob/main/src/pipeline/Stage.hpp"
    click CON "https://github.com/coder-na-r1/pipeline/blob/main/src/connector/SPSCConnector.hpp"
    click DL "https://github.com/coder-na-r1/pipeline/blob/main/src/dLManager/DLManager.hpp"
    click MM "https://github.com/coder-na-r1/pipeline/blob/main/src/monitoringManager/MonitoringManager.hpp"
    click NTB "https://github.com/coder-na-r1/pipeline/blob/main/src/networkBroker/ZmqReceiverBroker.hpp"
    click NTM "https://github.com/coder-na-r1/pipeline/blob/main/src/networkManager/StandartReceiverNetworkManager.hpp"
    click P "https://github.com/coder-na-r1/pipeline/blob/main/src/parser/Parser.hpp"
    click DS "https://github.com/coder-na-r1/pipeline/blob/main/dataStream/DataStream.cpp"
    click CPS "https://github.com/coder-na-r1/pipeline/blob/main/dataStream/scheme/ndarray.capnp"
    click CDS "https://github.com/coder-na-r1/pipeline/tree/main/include/cds/"
    click JSON "https://github.com/coder-na-r1/pipeline/blob/main/include/json/json.hpp"
```
