How cscfi/hpc-container-wrapper works
Overall Project Type and Purpose
• Tykky is a containerization tool designed for HPC environments where installations (especially those using conda or virtual environments) need to be “wrapped” inside containers.
• Its main goal is to reduce excessive IO load and file count on parallel filesystems (like Lustre) by converting typical installations into a containerized squashfs filesystem.
• The tool is implemented using a mix of bash and python scripts.
• It supports several “frontends” (such as conda-containerize, pip-containerize, wrap-container, and wrap-install) that cater to different use cases.
File Structure to Identify Architectural Components
• The top-level items indicate different layers:
◦ "bin" – Contains command-line entry points (symlinks) that eventually point to the main containerize script in the frontends.
◦ "frontends" – Houses the python scripts that parse user input, enforce tool-specific defaults, and route execution based on the chosen mode.
◦ "configs" – Provides multiple YAML configuration files indicating different environments (e.g., local, lumi, mahti, puhti).
◦ "templates" – Contains shell templates (for conda, venv, and general wrapping) used during installation and container preparations.
◦ "tests" – Automated tests help validate behavior using example configurations and scripts.
◦ "etc" and "share" – Support auxiliary functions, bash completion, and helper scripts for environment activation and deactivation.
◦ "construct.py" and "cw_common.py" – Central python modules that probably merge user and default configs to generate environment files (e.g., _vars.sh) for further steps.
• Scripts like pre.sh, post.sh, generate_wrappers.sh, and create_inst.sh orchestrate the multi-step installation and wrapping process.
Main Components and Their Relationships
A. User Interface / Command Invocation
• bin directory entry points, which are symlinks to commands that kick off the process.
B. Frontends (Python Scripts)
• These modules parse CLI arguments, set tool-specific defaults, and decide which mode (conda-containerize, pip-containerize, wrap-container, or wrap-install) to activate.
C. Configuration Management
• YAML configurations from the configs directory (both default and user-provided) read/merged by construct.py.
• Generation of _vars.sh with environment variable definitions is central. This is then sourced by subsequent bash scripts.
D. Installation / Containerization Process (Bash Scripts)
• pre.sh: Prepares the environment by fetching/download/copying the container image.
• create_inst: Executes the installation within the container. It is responsible for creating a squashfs image by running the actual installation commands using the mounted container.
◦ During this stage, the installation either uses a full host mount or default, specified mounts (as per configuration).
• generate_wrappers: Produces the “wrapper” scripts that make installations appear conventional. Involves mounting adjustments, handling of virtual environments/conda environments, and setting up symlink layers.
• post.sh: Finalizes the process by copying build files to the final installation directory and archiving the used build files.
E. Template Engine
• Templates (from the templates folder) which provide reusable snippets for conda and venv based installation or container wrapping.
F. External Dependencies / Systems
• Container runtimes: Apptainer or Singularity. The tool hardcodes singularity commands, using options for loop mounting images and handling bind mounts.
• The system’s hosting environment (parallel filesystem details) which is central to the benefits of using Tykky.
G. Auxiliary Components
• Bash completion scripts and shell functions in etc/ and share/ which facilitate user interaction (e.g., the “tykky” command for activating and deactivating environments).
Architectural Patterns and Design Choices
• Modularized Design: Splitting logic between Python frontends (for configuration and argument parsing) and Bash scripts (for orchestration and container handling).
• Configuration-Driven Behavior: Using YAML configs to drive installation parameters and variable settings.
• Wrapper Generation: A notable technique to “mask” containerization so that user commands look like standard installations.
• Dependency on External Tools (Singularity/Apptainer) for container control (specifically for mounting and namespace isolation).
• Emphasis on loosely coupling frontends and core shell scripts for clarity and modularity.
• Each component or module should be clearly labeled (e.g., “CLI/Entry Points”, “Frontends”, “Configuration Merging (construct.py)”, “Container Orchestration (pre.sh, create_inst, generate_wrappers.sh, post.sh)”, “Templates”, “Bash Helper Functions”).
• From command invocation (via bin scripts) to the respective frontend.
• From frontend to configuration processing (construct.py).
• Then sequential execution through pre-installation, installation, wrapper generation, and post-installation steps.
• Shell scripts vs. Python modules vs. configuration files.
• Internal components (e.g., configuration merging) vs. external interactions (container runtime, host filesystem).
• To capture process isolation steps (highlighting the use of private mount namespaces and bind mounts) and note the dependency on singularity-specific commands.
• Finally, include annotations for architectural decisions (like “wrapper generation to expose containerized apps as native installations” and “configuration overrides prioritized from both user and default configs”).