# cooperative-computing-lab/cctools architecture

> Architecture diagram of the GitHub repository cooperative-computing-lab/cctools, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-19). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/cooperative-computing-lab/cctools
- Repository: https://github.com/cooperative-computing-lab/cctools

## Overview

a. Distributed computing core: Include the central executables such as makeflow (workflow engine), vine (distributed task manager), and work_queue (job and task queue manager). These components form the backbone of the distributed computations.
b. Batch job and workflow modules: Represent components like “batch_job,” “chirp,” “deltadb,” and “devel” projects that provide wrappers, resource management, and alternative execution environments.
c. Resource monitoring and system utilities: Modules such as resource_monitor, dttools, and filetrace provide system and resource monitoring; show these as monitoring/support subsystems.
d. Bindings & Client Interfaces: The “bindings” subdirectories across chirp, work_queue, poncho, and taskvine (with Python2, Python3, Perl, and even Golang bindings) illustrate multi-language client interfaces and APIs. Indicate that these interface modules allow external applications to integrate with the core distributed system.
e. Packaging, configuration, and documentation: Note the packaging directories (build-conda, build-docker, macports) and rich documentation (doc, manuals, readthedocs) which support deployment, maintenance, and developer onboarding.

b. How monitoring tools (resource_monitor, dttools) feed performance data back to the scheduling and job execution engines.
c. Flow of job definitions from workflow modules through task dispatchers and into node-level executables that interface with clusters or clouds.

##### Architectural Patterns and Design Principles
a. Modular and layered design: Emphasize a separation of concerns where the workflow engine, job execution engine, resource manager, and client bindings each form distinct modules.
c. Scalability and distributed processing: Note that the design caters to large-scale computations by supporting heterogenous execution media (clusters, clouds, grids) and that components are decoupled through well-defined interfaces (e.g., SWIG bindings, REST APIs where applicable).

##### Technologies and Platforms
a. Languages: Predominantly C for low-level performance-critical modules alongside Python, Perl, and Golang bindings.
b. Schedulers and resource managers: Integration points with external systems like Condor, SLURM, Torque, and AWS (as seen in examples in the “devel” directory).
c. Build and deployment: Use configuration files (Makefile, module.mk, configure scripts, environment.yml in conda) that drive the build and packaging process.

## Diagram (Mermaid)

```mermaid
graph TD
    subgraph "Bindings & Client Interfaces"
        CB["Chirp Bindings"]:::bindings
        WB["Work Queue Bindings"]:::bindings
        PB["Poncho Bindings"]:::bindings
        TB["Taskvine Bindings"]:::bindings
    end

    subgraph "Distributed Computing Core"
        MF["makeflow (Workflow Engine)"]:::core
        WQC["work_queue (Queue Manager)"]:::core
        VT["vine/task manager (Task Manager)"]:::core
    end

    subgraph "Batch Job & Workflow Modules"
        BJ["batch_job (Job Wrapper)"]:::batch
        CC["chirp (Core Module)"]:::batch
        DD["deltadb"]:::batch
        DV["devel (Execution Environment)"]:::batch
        FT["filetrace (File Tracing)"]:::batch
    end

    subgraph "Resource Monitoring & System Utilities"
        RM["resource_monitor (Monitoring)"]:::monitoring
        DT["dttools (Tools)"]:::monitoring
    end

    subgraph "Packaging, Configuration & Documentation"
        PK["Packaging & Build Tools"]:::packaging
        DC["Documentation"]:::packaging
    end

    Ex["External Schedulers (Condor/SLURM/Torque)"]:::external

    %% Bindings to Distributed Core
    CB -->|"submit"| MF
    WB -->|"submit"| WQC
    PB -->|"integrates"| WQC
    TB -->|"submit"| VT

    %% Batch Job & Workflow Modules to Distributed Core
    BJ -->|"jobs"| VT
    CC -->|"initiate"| MF
    DD -->|"data"| BJ
    DV -->|"wraps"| WQC
    FT -->|"logs"| DV

    %% Distributed Core to External Schedulers
    MF -->|"dispatch"| Ex
    WQC -->|"dispatch"| Ex
    VT -->|"dispatch"| Ex

    %% Resource Monitoring to Distributed Core
    RM -->|"feeds"| MF
    RM -->|"feeds"| WQC
    DT -->|"audits"| VT

    %% Packaging & Documentation influence
    PK -->|"configures"| MF
    PK -->|"configures"| WQC
    DC -->|"guides"| CB
    DC -->|"guides"| WB
    DC -->|"guides"| PB
    DC -->|"guides"| TB

    %% Click Events
    click MF "https://github.com/cooperative-computing-lab/cctools/tree/master/makeflow/"
    click WQC "https://github.com/cooperative-computing-lab/cctools/tree/master/work_queue/"
    click VT "https://github.com/cooperative-computing-lab/cctools/blob/master/batch_job/src/vine_factory.c"
    click BJ "https://github.com/cooperative-computing-lab/cctools/tree/master/batch_job/"
    click CC "https://github.com/cooperative-computing-lab/cctools/tree/master/chirp/"
    click DD "https://github.com/cooperative-computing-lab/cctools/tree/master/deltadb/"
    click DV "https://github.com/cooperative-computing-lab/cctools/tree/master/devel/"
    click FT "https://github.com/cooperative-computing-lab/cctools/tree/master/devel/filetrace/"
    click RM "https://github.com/cooperative-computing-lab/cctools/tree/master/resource_monitor/"
    click DT "https://github.com/cooperative-computing-lab/cctools/tree/master/dttools/"
    click CB "https://github.com/cooperative-computing-lab/cctools/tree/master/chirp/src/bindings/"
    click WB "https://github.com/cooperative-computing-lab/cctools/tree/master/work_queue/src/bindings/"
    click PB "https://github.com/cooperative-computing-lab/cctools/tree/master/poncho/src/"
    click TB "https://github.com/cooperative-computing-lab/cctools/tree/master/taskvine/src/bindings/"
    click PK "https://github.com/cooperative-computing-lab/cctools/tree/master/packaging/"
    click DC "https://github.com/cooperative-computing-lab/cctools/tree/master/doc/"

    %% Styles
    classDef bindings fill:#cce5ff,stroke:#0066cc,stroke-width:2px;
    classDef core fill:#e2f0cb,stroke:#7fb800,stroke-width:2px;
    classDef batch fill:#fff2cc,stroke:#ff9900,stroke-width:2px;
    classDef monitoring fill:#f8d7da,stroke:#d9534f,stroke-width:2px;
    classDef packaging fill:#d1ecf1,stroke:#0c5460,stroke-width:2px;
    classDef external fill:#eaeaea,stroke:#666666,stroke-width:2px,fill-opacity:0.8;
```
