# cscfi/hpc-container-wrapper architecture

> Architecture diagram of the GitHub repository cscfi/hpc-container-wrapper, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-30). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/cscfi/hpc-container-wrapper
- Repository: https://github.com/cscfi/hpc-container-wrapper

## Overview

##### Overall Project Type and Purpose
- Tykky is a containerization tool designed for HPC environments where installations (especially those using conda or virtual environments) need to be “wrapped” inside containers.
- Its main goal is to reduce excessive IO load and file count on parallel filesystems (like Lustre) by converting typical installations into a containerized squashfs filesystem.
- The tool is implemented using a mix of bash and python scripts.
- It supports several “frontends” (such as conda-containerize, pip-containerize, wrap-container, and wrap-install) that cater to different use cases.

##### File Structure to Identify Architectural Components
- The top-level items indicate different layers:
◦ "bin" – Contains command-line entry points (symlinks) that eventually point to the main containerize script in the frontends.
◦ "frontends" – Houses the python scripts that parse user input, enforce tool-specific defaults, and route execution based on the chosen mode.
◦ "configs" – Provides multiple YAML configuration files indicating different environments (e.g., local, lumi, mahti, puhti).
◦ "templates" – Contains shell templates (for conda, venv, and general wrapping) used during installation and container preparations.
◦ "tests" – Automated tests help validate behavior using example configurations and scripts.
◦ "etc" and "share" – Support auxiliary functions, bash completion, and helper scripts for environment activation and deactivation.
◦ "construct.py" and "cw_common.py" – Central python modules that probably merge user and default configs to generate environment files (e.g., _vars.sh) for further steps.
- Scripts like pre.sh, post.sh, generate_wrappers.sh, and create_inst.sh orchestrate the multi-step installation and wrapping process.

##### Main Components and Their Relationships
A. User Interface / Command Invocation
- bin directory entry points, which are symlinks to commands that kick off the process.
B. Frontends (Python Scripts)
- These modules parse CLI arguments, set tool-specific defaults, and decide which mode (conda-containerize, pip-containerize, wrap-container, or wrap-install) to activate.
C. Configuration Management
- YAML configurations from the configs directory (both default and user-provided) read/merged by construct.py.
- Generation of _vars.sh with environment variable definitions is central. This is then sourced by subsequent bash scripts.
D. Installation / Containerization Process (Bash Scripts)
- pre.sh: Prepares the environment by fetching/download/copying the container image.
- create_inst: Executes the installation within the container. It is responsible for creating a squashfs image by running the actual installation commands using the mounted container.
◦ During this stage, the installation either uses a full host mount or default, specified mounts (as per configuration).
- generate_wrappers: Produces the “wrapper” scripts that make installations appear conventional. Involves mounting adjustments, handling of virtual environments/conda environments, and setting up symlink layers.
- post.sh: Finalizes the process by copying build files to the final installation directory and archiving the used build files.
E. Template Engine
- Templates (from the templates folder) which provide reusable snippets for conda and venv based installation or container wrapping.
F. External Dependencies / Systems
- Container runtimes: Apptainer or Singularity. The tool hardcodes singularity commands, using options for loop mounting images and handling bind mounts.
- The system’s hosting environment (parallel filesystem details) which is central to the benefits of using Tykky.
G. Auxiliary Components
- Bash completion scripts and shell functions in etc/ and share/ which facilitate user interaction (e.g., the “tykky” command for activating and deactivating environments).

##### Architectural Patterns and Design Choices
- Modularized Design: Splitting logic between Python frontends (for configuration and argument parsing) and Bash scripts (for orchestration and container handling).
- Configuration-Driven Behavior: Using YAML configs to drive installation parameters and variable settings.
- Wrapper Generation: A notable technique to “mask” containerization so that user commands look like standard installations.
- Dependency on External Tools (Singularity/Apptainer) for container control (specifically for mounting and namespace isolation).
- Emphasis on loosely coupling frontends and core shell scripts for clarity and modularity.

- Each component or module should be clearly labeled (e.g., “CLI/Entry Points”, “Frontends”, “Configuration Merging (construct.py)”, “Container Orchestration (pre.sh, create_inst, generate_wrappers.sh, post.sh)”, “Templates”, “Bash Helper Functions”).
- From command invocation (via bin scripts) to the respective frontend.
- From frontend to configuration processing (construct.py).
- Then sequential execution through pre-installation, installation, wrapper generation, and post-installation steps.
- Shell scripts vs. Python modules vs. configuration files.
- Internal components (e.g., configuration merging) vs. external interactions (container runtime, host filesystem).
- To capture process isolation steps (highlighting the use of private mount namespaces and bind mounts) and note the dependency on singularity-specific commands.
- Finally, include annotations for architectural decisions (like “wrapper generation to expose containerized apps as native installations” and “configuration overrides prioritized from both user and default configs”).

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% User Interface / CLI Entry Points
    subgraph "CLI/Entry Points"
        CLI1["CLI: conda-containerize"]:::cli
        CLI2["CLI: pip-containerize"]:::cli
        CLI3["CLI: wrap-container"]:::cli
        CLI4["CLI: wrap-install"]:::cli
    end

    %% Frontends (Python Scripts)
    subgraph "Frontends (Python Scripts)"
        F1["conda-containerize.py"]:::python
        F2["pip-containerize.py"]:::python
        F3["wrap-container.py"]:::python
        F4["wrap-install.py"]:::python
        F5["containerize (script)"]:::python
        F6["script_shared.py"]:::python
    end

    %% Configuration Management
    subgraph "Configuration Management"
        C1["YAML Configurations"]:::config
        C2["construct.py"]:::config
        C3["cw_common.py"]:::config
        C4["Merged Configs (_vars.sh)"]:::config
    end

    %% Container Orchestration (Shell Scripts)
    subgraph "Container Orchestration"
        O1["Pre-installation (pre.sh)"]:::shell
        O2["Installation (create_inst.sh)"]:::shell
        O3["Wrapper Generation (generate_wrappers.sh)"]:::shell
        O4["Post-installation (post.sh)"]:::shell
    end

    %% Template Engine
    subgraph "Template Engine"
        T1["Shell Templates"]:::template
    end

    %% Auxiliary Components
    subgraph "Auxiliary Components"
        A1["Bash Helper Scripts (etc)"]:::aux
        A2["Shell Functions (share)"]:::aux
    end

    %% External Dependencies / Systems
    subgraph "External Dependencies"
        E1["Container Runtimes (Singularity/Apptainer)"]:::external
        E2["Parallel Filesystem (Lustre, etc.)"]:::external
    end

    %% Connections from CLI to Frontends (each CLI maps to a corresponding frontend)
    CLI1 -->|"triggers"| F1
    CLI2 -->|"triggers"| F2
    CLI3 -->|"triggers"| F3
    CLI4 -->|"triggers"| F4

    %% Frontends flow into configuration management
    F1 -->|"processes args"| C1
    F2 -->|"processes args"| C1
    F3 -->|"processes args"| C1
    F4 -->|"processes args"| C1
    F5 -->|"shared utilities"| C1
    F6 -->|"shared utilities"| C1

    %% YAML configurations feed into config merging modules
    C1 -->|"loaded configs"| C2
    C1 -->|"loaded configs"| C3

    %% Merging of configurations
    C2 -->|"merges into"| C4
    C3 -->|"merges into"| C4

    %% Merged config flows into container orchestration steps
    C4 -->|"init environment"| O1
    O1 -->|"prepare container & call"| O2
    O2 -->|"install inside container"| O3
    O3 -->|"generate command wrappers"| O4

    %% Integration with Template Engine
    C4 -->|"uses templates"| T1

    %% Interaction with external systems
    O1 -->|"invokes"| E1
    O2 -->|"mounts on"| E2

    %% Post-install integration with auxiliary components
    O4 -->|"finalizes installation"| A1
    O4 -->|"finalizes installation"| A2

    %% Feedback loop: Auxiliary components enhance CLI user experience
    A1 -->|"shell integration"| CLI1
    A2 -->|"activation functions"| CLI1

    %% Click Events for CLI/Entry Points
    click CLI1 "https://github.com/cscfi/hpc-container-wrapper/tree/master/bin/conda-containerize"
    click CLI2 "https://github.com/cscfi/hpc-container-wrapper/tree/master/bin/pip-containerize"
    click CLI3 "https://github.com/cscfi/hpc-container-wrapper/tree/master/bin/wrap-container"
    click CLI4 "https://github.com/cscfi/hpc-container-wrapper/tree/master/bin/wrap-install"

    %% Click Events for Frontends
    click F1 "https://github.com/cscfi/hpc-container-wrapper/blob/master/frontends/conda-containerize.py"
    click F2 "https://github.com/cscfi/hpc-container-wrapper/blob/master/frontends/pip-containerize.py"
    click F3 "https://github.com/cscfi/hpc-container-wrapper/blob/master/frontends/wrap-container.py"
    click F4 "https://github.com/cscfi/hpc-container-wrapper/blob/master/frontends/wrap-install.py"
    click F5 "https://github.com/cscfi/hpc-container-wrapper/tree/master/frontends/containerize"
    click F6 "https://github.com/cscfi/hpc-container-wrapper/blob/master/frontends/script_shared.py"

    %% Click Events for Configuration Management
    click C1 "https://github.com/cscfi/hpc-container-wrapper/tree/master/configs/"
    click C2 "https://github.com/cscfi/hpc-container-wrapper/blob/master/construct.py"
    click C3 "https://github.com/cscfi/hpc-container-wrapper/blob/master/cw_common.py"

    %% Click Events for Container Orchestration
    click O1 "https://github.com/cscfi/hpc-container-wrapper/blob/master/pre.sh"
    click O2 "https://github.com/cscfi/hpc-container-wrapper/blob/master/create_inst.sh"
    click O3 "https://github.com/cscfi/hpc-container-wrapper/blob/master/generate_wrappers.sh"
    click O4 "https://github.com/cscfi/hpc-container-wrapper/blob/master/post.sh"

    %% Click Event for Template Engine
    click T1 "https://github.com/cscfi/hpc-container-wrapper/tree/master/templates/"

    %% Click Events for Auxiliary Components
    click A1 "https://github.com/cscfi/hpc-container-wrapper/tree/master/etc/"
    click A2 "https://github.com/cscfi/hpc-container-wrapper/tree/master/share/sh_functions/"

    %% Styles
    classDef cli fill:#FFDAB9,stroke:#FF8C00,stroke-width:2px;
    classDef python fill:#E0F7FA,stroke:#00796B,stroke-width:2px;
    classDef config fill:#E8EAF6,stroke:#3F51B5,stroke-width:2px;
    classDef shell fill:#FFF9C4,stroke:#FBC02D,stroke-width:2px;
    classDef template fill:#FFE0B2,stroke:#FB8C00,stroke-width:2px;
    classDef aux fill:#C8E6C9,stroke:#388E3C,stroke-width:2px;
    classDef external fill:#F0F4C3,stroke:#9E9D24,stroke-width:2px;
```
