# saforem2/ezpz architecture

> Architecture diagram of the GitHub repository saforem2/ezpz, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-03). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/saforem2/ezpz
- Repository: https://github.com/saforem2/ezpz

## Overview

##### Project Type and Purpose
- The project is a Python library/tool named “ezpz” that appears to facilitate launching and managing Python modules with a “setup environment” step, integration of configuration management (with YAML, JSON, and Hydra), logging, and support for various execution modes (including distributed or parallel execution).
- The README shows example commands for setting up the environment, installing via pip directly from Git, and launching modules—indicating that this project is designed to be integrated as a library in user projects as well as providing utilities for running jobs/modules in an automated manner.

##### File Structure to Pinpoint Key Components
- Core Library Source: The entire code lives under src/ezpz. This is where the major modules and utilities reside.
- Entry Points and CLI:
- Files like __main__.py and launch.py show that the project supports command-line launching of modules.
- The ezpz/bin directory contains shell scripts (affinity.sh, utils.sh) used to set up and manage the shell environment.
- Configuration Management:
- The conf directory holds various configuration files such as config.yaml, ds_config.[json|yaml], and specialized config segments for Hydra (including job_logging setups).
- This indicates a strong emphasis on dynamic configuration via YAML and possibly Hydra’s runtime configuration management.
- Logging Mechanism:
- The log subfolder contains its own configuration (subfolder for hydra logging settings) along with modules like console.py, handler.py, and style.py to manage how logs are formatted, displayed, and tested.
- Core Functional Modules:
- The modules like train.py, trainer.py, hf_trainer.py, and model.py suggest that the library has components for training or executing jobs.
- Additional utilities such as utils.py, jobs.py, runtime.py, and profile.py indicate support for handling job execution, benchmarking, and environment-specific tasks.
- Support for Distributed Execution:
- Modules like dist.py, pbs.py, and the inclusion of affinity settings in shell scripts point to built-in support for managing distributed or parallel tasks.

##### Main Components
a. Environment Setup & CLI Layer:
- A node for the command-line interface allowing users to invoke the tool (using launch.py and __main__.py).

b. Configuration Management Subsystem:
- Configuration files grouped under the conf directory.
- Nodes for static configuration (config.yaml, ds_config.json/yaml) and dynamic Hydra configurations (hydra job_logging, logdir settings).
- How these configurations feed into the other core components.

c. Logging Subsystem:
- Dependencies on configuration files (from both conf and log/conf) that determine logging behavior.

d. Core Functional Components:
- Nodes/modules that handle core functionalities—like job management, training (trainer.py, train.py, hf_trainer.py), and model management.
- “dist” and “pbs” modules that provide support for distributed computing.
- Supportive nodes for utilities (configs.py, utils.py), history management, plotting, and profiling.

e. External Integration & Deployment:
- Although not a separate microservice architecture, the installation steps and pip repository indicate external integration.
- How the library interacts as a standalone tool as well as an integrated dependency in broader systems.

##### Component Relationships and Data Flow
- Logging subsystem to all core modules, since logging is a cross-cutting concern.

##### Architectural Patterns and Design Principles
- Separation of concerns:
- There’s a clear separation between environment setup, configuration management, logging, and functional execution.
- Use of configuration-driven behavior (via YAML, JSON, and Hydra).
- That the system is built in a modular way, enabling individual parts (like logging and job launching) to be updated or extended independently.
- Infrastructure/Environment (shell scripts, CLI components)
- Configuration (YAML/JSON files and Hydra configurations)
- Core Business Logic (training, job management, distributed execution)
- Cross-Cutting Concerns (logging)

- Each component should have a clear, descriptive label.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% External Integration
    external["Pip Installation"]

    %% Environment Setup & CLI Layer
    subgraph "Environment Setup & CLI Layer"
        cli_main["CLI: __main__.py"]
        cli_launch["CLI: launch.py"]
        shell_affinity["Shell: affinity.sh"]
        shell_utils["Shell: utils.sh"]
    end

    %% Configuration Management Subsystem
    subgraph "Configuration Management Subsystem"
        config_yaml["Static Config: config.yaml"]
        config_ds_json["Static Config: ds_config.json"]
        config_ds_yaml["Static Config: ds_config.yaml"]
        hydra_colorlog["Hydra Config: colorlog1.yaml"]
        hydra_custom["Hydra Config: custom.yaml"]
        hydra_enrich["Hydra Config: enrich.yaml"]
        logdir_default["Logdir Config: default.yaml"]
        config_utils["Config Utils: configs.py"]
    end

    %% Logging Subsystem
    subgraph "Logging Subsystem"
        log_console["Logging Module: console.py"]
        log_handler["Logging Module: handler.py"]
        log_style["Logging Module: style.py"]
        log_config["Logging Module: config.py"]
        log_test["Logging Test: test.py"]
        log_test_log["Logging Test: test_log.py"]
        log_hydra_enrich["Logging Config: enrich.yaml"]
    end

    %% Core Functional Components
    subgraph "Core Functional Components"
        core_train["Job/Training: train.py"]
        core_trainer["Job/Training: trainer.py"]
        core_hf_trainer["Job/Training: hf_trainer.py"]
        core_jobs["Job: jobs.py"]
        core_model["Model: model.py"]
        core_loadjobenv["Env Mgt: loadjobenv.py"]
        core_savejobenv["Env Mgt: savejobenv.py"]
        core_utils["Utility: utils.py"]
        core_runtime["Runtime: runtime.py"]
        core_profile["Profile: profile.py"]
        core_history["History: history.py"]
        core_plot["Plot: plot.py"]
        core_cria["Additional Module: cria.py"]
    end

    %% Distributed Execution Support
    subgraph "Distributed Execution Support"
        dist_dist["Distributed Module: dist.py"]
        dist_pbs["Distributed Module: pbs.py"]
    end

    %% Auxiliary Components
    subgraph "Auxiliary Components"
        aux_test["Aux Test: test.py"]
        aux_test_dist["Aux Test: test_dist.py"]
        aux_tp["Technical Pipeline: tp"]
    end

    %% Relationships / Data Flow
    external -->|"triggers"| cli_main
    cli_launch -->|"loads"| config_yaml
    config_yaml -->|"configures"| core_train
    config_ds_yaml -->|"influences"| core_trainer
    hydra_enrich -->|"feeds"| log_hydra_enrich
    config_utils -->|"supports"| core_utils
    log_console -->|"logs for"| core_trainer
    log_config -->|"reports to"| core_trainer
    cli_launch -->|"initiates"| dist_dist
    shell_affinity -->|"sets affinity for"| dist_dist
    core_jobs -->|"tested by"| aux_test
    core_jobs -->|"validated by"| aux_test_dist
    core_utils -->|"assists"| aux_tp

    %% Click Events - Environment Setup & CLI Layer
    click cli_main "https://github.com/saforem2/ezpz/blob/main/src/ezpz/__main__.py"
    click cli_launch "https://github.com/saforem2/ezpz/blob/main/src/ezpz/launch.py"
    click shell_affinity "https://github.com/saforem2/ezpz/blob/main/src/ezpz/bin/affinity.sh"
    click shell_utils "https://github.com/saforem2/ezpz/blob/main/src/ezpz/bin/utils.sh"

    %% Click Events - Configuration Management Subsystem
    click config_yaml "https://github.com/saforem2/ezpz/blob/main/src/ezpz/conf/config.yaml"
    click config_ds_json "https://github.com/saforem2/ezpz/blob/main/src/ezpz/conf/ds_config.json"
    click config_ds_yaml "https://github.com/saforem2/ezpz/blob/main/src/ezpz/conf/ds_config.yaml"
    click hydra_colorlog "https://github.com/saforem2/ezpz/blob/main/src/ezpz/conf/hydra/job_logging/colorlog1.yaml"
    click hydra_custom "https://github.com/saforem2/ezpz/blob/main/src/ezpz/conf/hydra/job_logging/custom.yaml"
    click hydra_enrich "https://github.com/saforem2/ezpz/blob/main/src/ezpz/conf/hydra/job_logging/enrich.yaml"
    click logdir_default "https://github.com/saforem2/ezpz/blob/main/src/ezpz/conf/logdir/default.yaml"
    click config_utils "https://github.com/saforem2/ezpz/blob/main/src/ezpz/configs.py"

    %% Click Events - Logging Subsystem
    click log_console "https://github.com/saforem2/ezpz/blob/main/src/ezpz/log/console.py"
    click log_handler "https://github.com/saforem2/ezpz/blob/main/src/ezpz/log/handler.py"
    click log_style "https://github.com/saforem2/ezpz/blob/main/src/ezpz/log/style.py"
    click log_config "https://github.com/saforem2/ezpz/blob/main/src/ezpz/log/config.py"
    click log_test "https://github.com/saforem2/ezpz/blob/main/src/ezpz/log/test.py"
    click log_test_log "https://github.com/saforem2/ezpz/blob/main/src/ezpz/log/test_log.py"
    click log_hydra_enrich "https://github.com/saforem2/ezpz/blob/main/src/ezpz/log/conf/hydra/job_logging/enrich.yaml"

    %% Click Events - Core Functional Components
    click core_train "https://github.com/saforem2/ezpz/blob/main/src/ezpz/train.py"
    click core_trainer "https://github.com/saforem2/ezpz/blob/main/src/ezpz/trainer.py"
    click core_hf_trainer "https://github.com/saforem2/ezpz/blob/main/src/ezpz/hf_trainer.py"
    click core_jobs "https://github.com/saforem2/ezpz/blob/main/src/ezpz/jobs.py"
    click core_model "https://github.com/saforem2/ezpz/blob/main/src/ezpz/model.py"
    click core_loadjobenv "https://github.com/saforem2/ezpz/blob/main/src/ezpz/loadjobenv.py"
    click core_savejobenv "https://github.com/saforem2/ezpz/blob/main/src/ezpz/savejobenv.py"
    click core_utils "https://github.com/saforem2/ezpz/blob/main/src/ezpz/utils.py"
    click core_runtime "https://github.com/saforem2/ezpz/blob/main/src/ezpz/runtime.py"
    click core_profile "https://github.com/saforem2/ezpz/blob/main/src/ezpz/profile.py"
    click core_history "https://github.com/saforem2/ezpz/blob/main/src/ezpz/history.py"
    click core_plot "https://github.com/saforem2/ezpz/blob/main/src/ezpz/plot.py"
    click core_cria "https://github.com/saforem2/ezpz/blob/main/src/ezpz/cria.py"

    %% Click Events - Distributed Execution Support
    click dist_dist "https://github.com/saforem2/ezpz/blob/main/src/ezpz/dist.py"
    click dist_pbs "https://github.com/saforem2/ezpz/blob/main/src/ezpz/pbs.py"

    %% Click Events - Auxiliary Components
    click aux_test "https://github.com/saforem2/ezpz/blob/main/src/ezpz/test.py"
    click aux_test_dist "https://github.com/saforem2/ezpz/blob/main/src/ezpz/test_dist.py"
    click aux_tp "https://github.com/saforem2/ezpz/tree/main/src/ezpz/tp"

    %% Styles
    classDef cli fill:#AED6F1,stroke:#333,stroke-width:2px;
    classDef config fill:#A9DFBF,stroke:#333,stroke-width:2px;
    classDef log fill:#F9E79F,stroke:#333,stroke-width:2px;
    classDef core fill:#FAD7A0,stroke:#333,stroke-width:2px;
    classDef dist fill:#D2B4DE,stroke:#333,stroke-width:2px;
    classDef aux fill:#F5B7B1,stroke:#333,stroke-width:2px;
    classDef external fill:#D5DBDB,stroke:#333,stroke-width:2px;

    class cli_main,cli_launch,shell_affinity,shell_utils cli;
    class config_yaml,config_ds_json,config_ds_yaml,hydra_colorlog,hydra_custom,hydra_enrich,logdir_default,config_utils config;
    class log_console,log_handler,log_style,log_config,log_test,log_test_log,log_hydra_enrich log;
    class core_train,core_trainer,core_hf_trainer,core_jobs,core_model,core_loadjobenv,core_savejobenv,core_utils,core_runtime,core_profile,core_history,core_plot,core_cria core;
    class dist_dist,dist_pbs dist;
    class aux_test,aux_test_dist,aux_tp aux;
    class external external;
```
